testdriverai 5.2.1 → 5.3.0

package/docs/action/browser.mdx

@@ -0,0 +1,129 @@
+---
+title: "Testing Multiple Browsers"
+sidebarTitle: "Browsers"
+description: "Configure prerun scripts to launch different browsers in TestDriver."
+---
+
+## Overview
+Prerun scripts allow you to configure the TestDriver virtual machine (VM) to launch specific browsers before running your tests. This is particularly useful for testing your application across multiple browsers and ensuring compatibility.
+
+By using prerun scripts, you can:
+- Install and launch different browsers based on your test requirements.
+- Customize the test environment for specific scenarios.
+- Ensure consistent browser configurations across operating systems.
+
+---
+
+## Supported Browsers
+The following browsers can be installed and launched using prerun scripts:
+- **Google Chrome**
+- **Mozilla Firefox**
+
+---
+
+## Prerun Script Example: Installing and Launching Browsers
+The following example demonstrates how to install and launch Google Chrome or Firefox on Windows, macOS, and Linux using a prerun script.
+
+### Example: Prerun Script
+```yaml
+prerun: |
+  if [ "${{ matrix.browser }}" == "chrome" ]; then
+    if [ "${{ matrix.os }}" == "windows" ]; then
+      # Install and launch Google Chrome on Windows
+      $ProgressPreference = 'SilentlyContinue'
+      Invoke-WebRequest -Uri "https://dl.google.com/chrome/install/latest/chrome_installer.exe" -OutFile "$env:TEMP\chrome_installer.exe"
+      Start-Process -FilePath "$env:TEMP\chrome_installer.exe" -ArgumentList "/silent", "/install" -Wait
+      Start-Process -FilePath "C:\Program Files\Google\Chrome\Application\chrome.exe"
+    elif [ "${{ matrix.os }}" == "mac" ]; then
+      # Install and launch Google Chrome on macOS
+      brew install --cask google-chrome
+      open -a "Google Chrome"
+    else
+      # Install and launch Google Chrome on Linux
+      wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
+      sudo apt install ./google-chrome-stable_current_amd64.deb -y
+      google-chrome &
+    fi
+  else
+    if [ "${{ matrix.os }}" == "windows" ]; then
+      # Install and launch Firefox on Windows
+      $ProgressPreference = 'SilentlyContinue'
+      Invoke-WebRequest -Uri "https://download.mozilla.org/?product=firefox-latest&os=win64&lang=en-US" -OutFile "$env:TEMP\firefox_installer.exe"
+      Start-Process -FilePath "$env:TEMP\firefox_installer.exe" -ArgumentList "/S" -Wait
+      Start-Process -FilePath "C:\Program Files\Mozilla Firefox\firefox.exe"
+    elif [ "${{ matrix.os }}" == "mac" ]; then
+      # Install and launch Firefox on macOS
+      brew install --cask firefox
+      open -a "Firefox"
+    else
+      # Install and launch Firefox on Linux
+      sudo apt update
+      sudo apt install firefox -y
+      firefox &
+    fi
+  fi
+```
+
+---
+
+## Testing Multiple Browsers
+You can use the GitHub matrix strategy to test your application across multiple browsers. This ensures comprehensive browser compatibility testing.
+
+### Example: Matrix Strategy for Browser Testing
+```yaml
+strategy:
+  matrix:
+    os: [windows, mac, linux]
+    browser: [chrome, firefox]
+```
+
+### Full Workflow Example
+```yaml
+name: Test Action
+
+permissions:
+  actions: read
+  contents: read
+  statuses: write
+  pull-requests: write
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
+
+jobs:
+  test-action:
+    name: Test Action
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        os: [windows, mac, linux]
+        browser: [chrome, firefox]
+    steps:
+      - name: Set up Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: '16'
+
+      - uses: replayableio/testdriver-action@main
+        with:
+          prompt: |
+            1. open youtube
+            2. find a cat video
+            3. quit the browser
+            4. /summarize
+          os: ${{ matrix.os }}
+          prerun: |
+            # Add prerun script here
+          key: ${{ secrets.TESTDRIVER_API_KEY }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          FORCE_COLOR: "3"
+```
+
+---
+
+## Notes
+- **Cross-Platform Compatibility**: Ensure your prerun scripts are compatible with the operating system specified in the matrix.
+- **Browser Versions**: Always install the latest stable versions of browsers to ensure compatibility with modern web standards.
+- **Performance**: Launching browsers in prerun scripts ensures they are ready for immediate use during tests, reducing setup time.

package/docs/action/os.mdx

@@ -0,0 +1,157 @@
+---
+title: "Testing Multiple Operating Systems"
+sidebarTitle: "Operating Systems"
+description: "Configuring Operating Systems in TestDriver GitHub Action"
+---
+
+## Overview
+The TestDriver GitHub Action allows you to run tests on multiple operating systems, enabling cross-platform compatibility testing. By configuring the `os` parameter, you can specify the operating system for your test environment. This flexibility ensures that your application behaves consistently across platforms.
+
+---
+
+## Supported Operating Systems
+The following operating systems are currently supported:
+
+| OS       | Version                  | Instance Type | Architecture   | Notes                          |
+|----------|--------------------------|---------------|----------------|--------------------------------|
+| `windows`| Windows Server 2022 Base| `t2.large`    | 64-bit (x86)   | Available to all users.        |
+| `mac`    | macOS Sonoma             | `mac1.metal`  | x86_64_mac     | Available to Enterprise users. |
+| `linux`  | Ubuntu 20.04 LTS         | `t2.large`    | 64-bit (x86)   | Default for most workflows.    |
+
+---
+
+## Configuring the Operating System
+To specify the operating system, use the `os` parameter in the GitHub Action configuration. For example:
+
+### Example: Running a Test on Windows
+```yaml
+with:
+  os: windows
+```
+
+### Example: Running a Test on macOS
+```yaml
+with:
+  os: mac
+```
+
+### Example: Running a Test on Linux
+```yaml
+with:
+  os: linux
+```
+
+---
+
+## Prerun Scripts and OS-Specific Commands
+Prerun scripts are executed on the specified operating system. The scripting language depends on the OS:
+- **Windows**: Use PowerShell.
+- **macOS**: Use Bash.
+- **Linux**: Use Bash.
+
+### Example: Installing Browsers Based on OS
+The following example demonstrates how to install Google Chrome or Firefox on Windows, macOS, and Linux using a prerun script:
+
+```yaml
+prerun: |
+  if [ "${{ matrix.browser }}" == "chrome" ]; then
+    if [ "${{ matrix.os }}" == "windows" ]; then
+      # Install Google Chrome on Windows
+      $ProgressPreference = 'SilentlyContinue'
+      Invoke-WebRequest -Uri "https://dl.google.com/chrome/install/latest/chrome_installer.exe" -OutFile "$env:TEMP\chrome_installer.exe"
+      Start-Process -FilePath "$env:TEMP\chrome_installer.exe" -ArgumentList "/silent", "/install" -Wait
+    elif [ "${{ matrix.os }}" == "mac" ]; then
+      # Install Google Chrome on macOS
+      brew install --cask google-chrome
+    else
+      # Install Google Chrome on Linux
+      wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
+      sudo apt install ./google-chrome-stable_current_amd64.deb -y
+    fi
+  else
+    if [ "${{ matrix.os }}" == "windows" ]; then
+      # Install Firefox on Windows
+      $ProgressPreference = 'SilentlyContinue'
+      Invoke-WebRequest -Uri "https://download.mozilla.org/?product=firefox-latest&os=win64&lang=en-US" -OutFile "$env:TEMP\firefox_installer.exe"
+      Start-Process -FilePath "$env:TEMP\firefox_installer.exe" -ArgumentList "/S" -Wait
+    elif [ "${{ matrix.os }}" == "mac" ]; then
+      # Install Firefox on macOS
+      brew install --cask firefox
+    else
+      # Install Firefox on Linux
+      sudo apt update
+      sudo apt install firefox -y
+    fi
+  fi
+```
+
+---
+
+## Testing Multiple Operating Systems and Browsers
+You can use the GitHub matrix strategy to test your application across multiple operating systems and browsers. This ensures comprehensive coverage for your tests.
+
+### Example: Matrix Strategy for OS and Browser Testing
+```yaml
+strategy:
+  matrix:
+    os: [windows, mac, linux]
+    browser: [chrome, firefox]
+```
+
+### Full Workflow Example
+```yaml
+name: Test Action
+
+permissions:
+  actions: read
+  contents: read
+  statuses: write
+  pull-requests: write
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
+
+jobs:
+  test-action:
+    name: Test Action
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        os: [windows, mac, linux]
+        browser: [chrome, firefox]
+    steps:
+      - name: Set up Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: '16'
+
+      - name: Install Dashcam Chrome
+        run: |
+          npm init -y
+          npm install dashcam-chrome
+
+      - uses: replayableio/testdriver-action@main
+        with:
+          prompt: |
+            1. open youtube
+            2. find a cat video
+            3. quit the browser
+            4. /summarize
+          os: ${{ matrix.os }}
+          prerun: |
+            # Add prerun script here
+          key: ${{ secrets.TESTDRIVER_API_KEY }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          FORCE_COLOR: "3"
+```
+
+---
+
+## Notes
+- **macOS Availability**: macOS testing is only available to Enterprise customers.
+- **Prerun Scripts**: Ensure your prerun scripts are compatible with the specified operating system.
+- **Cross-Platform Testing**: Use the matrix strategy to test across multiple OS and browser combinations for maximum coverage.
+- **Linux Default**: Linux is the default operating system for most workflows and is ideal for lightweight, fast testing.
+```

package/docs/action/output.mdx

@@ -0,0 +1,98 @@
+---
+title: "GitHub Action Output"
+sidebarTitle: "Action Output"
+description: "Understanding and utilizing the output variables from the TestDriver GitHub Action"
+---
+
+## Overview
+The TestDriver GitHub Action provides several output variables that can be used to create powerful workflows by chaining actions together. These outputs allow you to post results as comments, send notifications, or integrate with third-party test reporting tools.
+
+---
+
+## Output Variables
+
+| Variable   | Description                                                                 |
+|------------|-----------------------------------------------------------------------------|
+| `summary`  | Contains the TestDriver AI text summary result of the action execution.     |
+| `link`     | A link to the Dashcam dashboard for debugging test runs.                   |
+| `markdown` | Markdown-formatted shareable link, including a screenshot of the desktop.  |
+| `success`  | Indicates whether the action passed successfully (`true` or `false`).      |
+
+---
+
+## Example: Creating a Comment on a Pull Request
+The following example demonstrates how to use the output variables to create a comment on a pull request after every TestDriver execution.
+
+### Workflow Example
+```yaml
+name: TestDriver.ai
+
+permissions:
+  actions: read
+  contents: read
+  statuses: write
+  pull-requests: write
+
+on:
+  pull_request:
+
+jobs:
+  test:
+    name: "TestDriver"
+    runs-on: ubuntu-latest
+    id: run-testdriver
+    steps:
+      - uses: dashcamio/testdriver@main
+        version: v4.0.0
+        key: ${{ secrets.TESTDRIVER_API_KEY }}
+        with:
+          prompt: |
+            1. /run /Users/ec2-user/actions-runner/_work/testdriver/testdriver/.testdriver/test.yml
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          FORCE_COLOR: "3"
+
+  - name: Create comment on PR
+    if: ${{ always() }}
+    uses: peter-evans/create-or-update-comment@v3
+    with:
+      issue-number: ${{ github.event.pull_request.number }}
+      body: |
+        **Test Summary:**
+        ${{ steps.run-testdriver.outputs.summary }}
+
+        **Markdown Report:**
+        ${{ steps.run-testdriver.outputs.markdown }}
+
+        **Dashcam Link:**
+        [View Test Results](${{ steps.run-testdriver.outputs.link }})
+```
+
+---
+
+## Use Cases for Output Variables
+
+### 1. Post Test Results as Comments
+Use the `summary` and `markdown` outputs to post detailed test results as comments on pull requests. This provides immediate feedback to developers.
+
+### 2. Send Notifications on Failure
+Use the `success` output to trigger notifications (e.g., email or Slack) when a test fails.
+
+Example:
+```yaml
+- name: Notify on Failure
+  if: ${{ steps.run-testdriver.outputs.success == 'false' }}
+  run: |
+    echo "Test failed! Sending notification..."
+    # Add your notification logic here
+```
+
+### 3. Integrate with Third-Party Tools
+Use the `link` output to upload test results to third-party test reporting tools or dashboards.
+
+---
+
+## Notes
+- The `link` output provides a direct URL to the Dashcam dashboard, making it easy to debug test runs.
+- The `markdown` output includes a screenshot of the desktop, which is useful for visualizing test results.
+- Always use the `success` output to handle conditional workflows based on test outcomes.

package/docs/action/performance.mdx

@@ -0,0 +1,71 @@
+---
+title: "Optimizing Performance in TestDriver"
+sidebarTitle: "Performance Optimization"
+description: "Optimizing Performance in TestDriver"
+---
+
+## Overview
+Optimizing your TestDriver tests can significantly reduce execution time, ensuring faster feedback for developers and smoother CI/CD workflows. While TestDriver's AI-powered capabilities are robust, using them efficiently is key to achieving the best performance.
+
+---
+
+## Tips for Improving Performance
+
+### 1. Use Parallel Testing
+Parallel testing allows you to split your test actions into multiple files and run them simultaneously. This can drastically reduce the total runtime of your test suite.
+
+#### How to Implement Parallel Testing
+- Divide your test steps into smaller, independent YAML files.
+- Use the `run` command or GitHub matrix strategy to execute these files in parallel.
+
+Example:
+```yaml
+strategy:
+  matrix:
+    test_file: 
+      - tests/login.yml
+      - tests/search.yml
+      - tests/cart.yml
+```
+
+---
+
+### 2. Use Optimized Matching Methods
+For actions like `hover-text`, `wait-for-text`, and `scroll-until-text`, use the `turbo` matching method instead of `ai`. The `turbo` method uses text similarity to quickly compute the most relevant match, making it about 40% faster than the `ai` method.
+
+#### Example:
+```yaml
+command: hover-text
+text: Sign In
+description: login button
+action: click
+method: turbo
+```
+
+---
+
+### 3. Use `async` Asserts
+The `assert` command supports the `async: true` property, allowing you to create non-blocking assertions. This means your tests can continue running while the assertion is being validated, saving valuable time.
+
+#### Example:
+```yaml
+command: assert
+expect: The user is logged in
+async: true
+```
+
+---
+
+## Best Practices
+- **Minimize AI Matching**: Use AI-powered matching methods only when necessary. For common actions, rely on optimized methods like `turbo`.
+- **Break Down Tests**: Split large, monolithic test files into smaller, focused tests to enable parallel execution.
+- **Leverage Asynchronous Features**: Use `async` properties wherever possible to avoid blocking test execution.
+- **Monitor Performance**: Regularly review test execution times and identify bottlenecks.
+
+---
+
+## Notes
+- Optimizing performance not only saves time but also reduces resource usage, making your CI/CD pipelines more efficient.
+- For large test suites, combining parallel testing with optimized matching methods can lead to significant time savings.
+- Always balance performance optimizations with test reliability to ensure accurate results.
+

package/docs/action/prerun.mdx

@@ -0,0 +1,80 @@
+---
+title: "Prerun Scripts"
+sidebarTitle: "Prerun Scripts"
+description: "Learn how to customize and set up your TestDriver environment to optimize your CI/CD pipeline."
+---
+
+## Overview
+Prerun scripts are commands executed on a TestDriver virtual machine (VM) before each test in a CI/CD pipeline. They are used to prepare the environment by provisioning the VM, installing dependencies, configuring settings, or building the application. This ensures a consistent and reproducible environment for every test execution.
+
+By using prerun scripts, you can:
+- Speed up test setup.
+- Prevent test failures caused by inconsistent environments.
+- Promote reproducible builds for reliable test results.
+
+---
+
+## Use Cases
+Prerun scripts are ideal for:
+- Installing necessary dependencies (e.g., browsers, libraries, or tools).
+- Building your application or running setup scripts.
+- Configuring the VM to match specific test requirements.
+- Preparing staging environments or accessing private resources.
+
+---
+
+## Example: Installing Arc Browser
+The following example demonstrates how to use a prerun script to download and install the Arc Browser on a Windows VM before running tests.
+
+```yaml
+# permissions and other setup here
+
+jobs:
+  test:
+    name: "TestDriver"
+    runs-on: ubuntu-latest
+    steps:
+      # Use the TestDriver GitHub Action
+      - uses: testdriverai/action@main
+        with:
+          prerun: |
+            # Get the IPv6 address of the VM
+            Get-NetIPAddress -AddressFamily IPv6
+            # URL for the Arc browser installer
+            $installerUrl = "https://releases.arc.net/windows/ArcInstaller.exe"
+            # Location to save the installer
+            $installerPath = "$env:USERPROFILE\Downloads\ArcInstaller.exe"
+            # Download the Arc browser installer
+            Write-Host "Downloading Arc browser installer..."
+            Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath
+            # Check if the download was successful
+            if (Test-Path $installerPath) {
+                Write-Host "Download successful. Running the installer..."
+                Start-Process -FilePath $installerPath -ArgumentList '/silent' -Wait
+                Start-Sleep -Seconds 10
+            } else {
+                Write-Host "Failed to download the Arc browser installer."
+            }
+```
+
+---
+
+## Key Points
+- **Provisioning**: Prerun scripts allow you to provision the VM with the necessary tools and configurations before running tests.
+- **Reproducibility**: By ensuring a consistent environment, prerun scripts help prevent flaky tests caused by environmental differences.
+- **Flexibility**: You can use prerun scripts to customize the VM for specific test scenarios, such as installing alternative browsers or setting up staging environments.
+
+---
+
+## Best Practices
+- **Keep It Simple**: Write clear and concise prerun scripts to minimize setup time and reduce complexity.
+- **Error Handling**: Include checks to verify that dependencies are installed successfully. Log errors to help debug issues.
+- **Optimize Performance**: Cache dependencies or use lightweight tools to speed up the setup process.
+- **Security**: Avoid hardcoding sensitive information in prerun scripts. Use GitHub secrets to securely pass credentials or tokens.
+
+---
+
+## Notes
+- Prerun scripts are executed on the VM before the test suite begins.
+- They are essential for ensuring a consistent and reliable test environment.
+- For advanced workflows, combine prerun scripts with TestDriver prompts to create dynamic and flexible test setups.

package/docs/action/secrets.mdx

@@ -0,0 +1,103 @@
+---
+title: "Managing Secrets in GitHub Actions"
+sidebarTitle: "Secrets Management"
+description: "Discover how to securely configure and optimize your TestDriver environment for seamless CI/CD workflows."
+---
+
+## Overview
+When using TestDriver to test your application, you may need to securely store and use sensitive information such as usernames, passwords, API keys, or other secrets. GitHub Actions provides a secure way to manage these secrets, ensuring they are not exposed in your test files or logs.
+
+---
+
+## Why Use Secrets?
+- **Security**: Secrets are encrypted and stored securely in your GitHub repository.
+- **Masking**: TestDriver automatically masks secrets in all test output, including debugging logs.
+- **Reusability**: Secrets can be reused across multiple workflows and test files.
+
+---
+
+## Step 1: Replace Hardcoded Secrets in Test Files
+To securely use secrets in your TestDriver test files, replace hardcoded values with placeholders in the format `${TD_YOUR_SECRET}`. TestDriver will parse and mask any secrets that begin with `TD_`.
+
+### Example Test File
+```yaml
+version: 4.1.35
+steps:
+  - prompt: sign in with username and password
+    commands:
+      - command: focus-application
+        name: Google Chrome
+      - command: hover-text
+        text: Email or phone
+        description: email input field
+        action: click
+      - command: type
+        text: ${TD_USERNAME}
+      - command: hover-text
+        text: Next
+        description: next button after entering email
+        action: click
+      - command: hover-text
+        text: Password
+        description: password input field
+        action: click
+      - command: type
+        text: ${TD_PASSWORD}
+```
+
+---
+
+## Step 2: Add Secrets to Your GitHub Repository
+1. Navigate to your GitHub repository.
+2. Go to **Settings** > **Secrets and variables** > **Actions**.
+3. Click **New repository secret**.
+4. Add your secrets (e.g., `TD_USERNAME`, `TD_PASSWORD`, `TESTDRIVER_API_KEY`).
+
+For detailed instructions, refer to the [GitHub Docs on using secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets).
+
+---
+
+## Step 3: Supply Secrets to GitHub Actions
+When running TestDriver tests via GitHub Actions, supply your secrets in the `env` section of the workflow file.
+
+### Example Workflow
+```yaml
+name: TestDriver Test
+
+permissions:
+  actions: read
+  contents: read
+  statuses: write
+  pull-requests: write
+
+jobs:
+  test:
+    name: "TestDriver"
+    runs-on: ubuntu-latest
+    steps:
+      - uses: testdriverai/action@main
+        with:
+          key: ${{ secrets.TESTDRIVER_API_KEY }}
+          prompt: | 
+            1. /run tests/signin.yml
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          TD_USERNAME: ${{ secrets.TD_USERNAME }}
+          TD_PASSWORD: ${{ secrets.TD_PASSWORD }}
+```
+
+---
+
+## Best Practices
+- **Use Descriptive Names**: Name your secrets clearly (e.g., `TD_USERNAME`, `TD_PASSWORD`) to make them easy to identify.
+- **Rotate Secrets Regularly**: Update your secrets periodically to enhance security.
+- **Limit Access**: Only provide access to secrets for workflows and team members that require them.
+- **Mask Secrets**: Ensure all secrets are masked in logs by using the `TD_` prefix.
+
+---
+
+## Notes
+- Secrets are encrypted and only accessible during the workflow run.
+- TestDriver automatically masks secrets in test output to prevent accidental exposure.
+- For additional security, avoid hardcoding sensitive information in your test files or workflows.
+