testdriverai 5.2.2 → 5.3.1

package/docs/getting-started/generating.mdx

@@ -0,0 +1,210 @@
+---
+title: "Generating Test Plans with TestDriver.ai"
+sidebarTitle: "Generating Tests"
+description: "Learn how to generate test plans locally with TestDriver.ai and integrate them into your testing workflow."
+---
+
+# Running Locally Generated Test Plans with TestDriver.ai and Managing Generated PRs
+
+This guide explains how to use **TestDriver.ai** to generate test plans locally, commit them to your repository, and run them in parallel using GitHub Actions. Additionally, it covers how **TestDriver.ai** can automatically create **pull requests (PRs)** for test results, enabling collaboration and review.
+
+---
+
+## Workflow Overview
+
+1. **Generate Test Plans Locally**: Use the `/generate` command to create test plans.
+2. **Commit Test Plans to the Repository**: Add the generated files to your repository for version control.
+3. **Run Tests in Parallel**: Use GitHub Actions to execute the committed test plans in parallel.
+4. **Create PRs for Test Results**: Automatically generate PRs for test results, allowing teams to review and merge updates.
+
+---
+
+## Step 1: Generate Test Plans Locally
+
+### Run `/generate` Locally
+
+1. Open your terminal and navigate to your project directory.
+2. Run the following command to start TestDriver.ai locally:
+
+
+```bash
+testdriverai
+> /generate web 10
+
+```
+
+This will generate 10 test plans for your website or application.
+
+---
+
+### Save the Generated Files
+
+TestDriver.ai will save the generated test plans as YAML files in the `testdriver/generate/` directory by default. For example:
+
+```
+testdriver/
+├── generate/
+│   ├── test_1.yml
+│   ├── test_2.yml
+│   ├── test_3.yml
+
+```
+
+---
+
+### Commit the Files to Your Repository
+
+1. Add the generated files to your Git repository:
+
+```bash
+git add testdriver/generate/
+git commit -m "Add locally generated test plans"
+git push origin main
+```
+
+---
+
+## Step 2: Run Tests in Parallel Using GitHub Actions
+
+### Workflow: **Run Locally Generated Tests**
+
+This workflow dynamically discovers the committed test files and runs them in parallel.
+
+#### Example Workflow: `.github/workflows/run-tests.yml`
+
+```yaml
+name: Run Locally Generated Test Plans
+
+on:
+  push:
+    paths:
+      - 'testdriver/generate/**'
+  workflow_dispatch:
+
+jobs:
+  gather-test-files:
+    name: Gather Test Files
+    runs-on: ubuntu-latest
+    outputs:
+      test_files: ${{ steps.test_list.outputs.files }}
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v2
+
+      - name: Find all test files
+        id: test_list
+        run: |
+          FILES=$(ls ./testdriver/generate/*.yml)
+          FILES_JSON=$(echo "$FILES" | jq -R -s -c 'split("\n")[:-1]')
+          echo "::set-output name=files::$FILES_JSON"
+
+  run-tests:
+    name: Run Tests in Parallel
+    needs: gather-test-files
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        test_file: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
+      fail-fast: false
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v2
+
+      - name: Run TestDriver.ai
+        uses: testdriverai/action@main
+        with:
+          key: ${{ secrets.TESTDRIVER_API_KEY }}
+          prompt: |
+            1. /run ${{ matrix.test_file }}
+          create-pr: true
+          pr-title: "TestDriver.ai / Test Results for ${{ matrix.test_file }}"
+          pr-branch: testdriver/results-${{ matrix.test_file }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          FORCE_COLOR: "3"
+
+```
+
+
+---
+
+### Workflow Steps Explained
+
+1. **Trigger on Push**:
+- The workflow triggers whenever files in the `testdriver/generate/` directory are pushed to the repository.
+
+2. **Gather Test Files**:
+- The `gather-test-files` job collects all YAML files in the `testdriver/generate/` directory and outputs them as a JSON array.
+
+3. **Run Tests in Parallel**:
+- The `run-tests` job uses the matrix strategy to run each test file in parallel.
+
+4. **Create PRs for Test Results**:
+- For each test file, a new branch is created with the test results, and a PR is opened for review.
+
+---
+
+## Step 3: Generated PRs for Test Results
+
+### How PRs Are Created
+
+When the tests are executed, TestDriver.ai automatically:
+1. Creates a new branch for each test result (e.g., `testdriver/results-test_1.yml`).
+2. Commits the test results to the branch.
+3. Opens a pull request with the test results for review.
+
+---
+
+### Example PR Details
+
+#### PR Title:
+`TestDriver.ai / Test Results for test_1.yml`
+
+#### PR Description:
+
+
+```
+### Test Results for test_1.yml
+
+**Test Summary**:
+- ✅ Step 1: Opened the homepage.
+- ✅ Step 2: Verified the login button is visible.
+- ❌ Step 3: Failed to submit the login form.
+
+**Details**:
+- Error: The "Submit" button was not clickable.
+
+Please review the test results and address any issues.
+
+```
+
+---
+
+### Benefits of Generated PRs
+
+1. **Collaboration**: Teams can review test results directly in GitHub and discuss issues in the PR comments.
+2. **Traceability**: Each test result is tied to a specific branch and PR, making it easy to track changes.
+3. **Continuous Improvement**: Failed tests can be addressed and merged back into the main branch after fixes.
+
+---
+
+## Example Output
+
+### GitHub Actions Dashboard:
+- ✅ **Run Tests in Parallel**: All tests passed.
+- ❌ **Run Tests in Parallel**: 1 test failed. View logs for details.
+
+### Pull Requests:
+- **PR #123**: `TestDriver.ai / Test Results for test_1.yml`
+- **PR #124**: `TestDriver.ai / Test Results for test_2.yml`
+
+---
+
+## Summary
+
+1. **Generate Tests Locally**: Use the `/generate` command to create test plans.
+2. **Commit and Run Tests**: Push the generated files to your repository and execute them in parallel.
+3. **Review Generated PRs**: Collaborate on test results using automatically created pull requests.
+4. **Merge Fixes**: Address issues and merge the PRs back into the main branch.
+
+By leveraging TestDriver.ai's ability to generate and manage PRs, you can streamline your testing workflow and improve collaboration across your team.

package/docs/getting-started/running.mdx

@@ -0,0 +1,67 @@
+---
+title: "Running Tests Locally with TestDriver.ai"
+sidebarTitle: "Running Tests"
+description: "Learn how to execute test plans locally using TestDriver.ai's `/run` command for debugging and validation."
+---
+
+# Running Tests Locally with `/run` in TestDriver.ai
+
+The `/run` command in TestDriver.ai allows you to execute previously created test plans locally. This is particularly useful for debugging, validating test scripts, or running tests in a controlled environment before committing them to a CI/CD pipeline.
+
+---
+
+## How to Use `/run`
+
+1. **Navigate to Your Test Directory**:
+Ensure you are in the directory where your test files are stored (e.g., `testdriver/`).
+
+2. **Run a Specific Test File**:
+Use the `/run` command followed by the path to the test file you want to execute. For example:
+
+```bash
+testdriverai
+> /run testdriver/test.yml
+```
+This will execute the test plan defined in `test.yml`.
+
+---
+
+## Benefits of Running Tests Locally
+
+1. **Quick Feedback**:
+- Test changes immediately without waiting for CI/CD pipelines.
+- Validate test scripts before committing them to version control.
+
+2. **Debugging**:
+- Identify and fix issues in test scripts or application behavior in a controlled environment.
+
+3. **Offline Testing**:
+- Run tests locally without requiring access to cloud-based runners or external environments.
+
+4. **Iterative Development**:
+- Modify test scripts and re-run them quickly to refine test cases.
+
+---
+
+## Example Workflow for Local Testing
+
+1. **Create a Test File**:
+Save the following YAML file as `testdriver/test.yml`:
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Open the homepage
+  - prompt: Verify the login button is visible
+  - prompt: Click the login button
+  - prompt: Verify the login form is displayed
+```
+
+2. **Run the Test Locally**:
+Execute the test using the `/run` command:
+
+```bash
+testdriverai
+> /run testdriver/test.yml   
+```
+

package/docs/getting-started/setup.mdx

@@ -0,0 +1,133 @@
+---
+title: "Setting Up TestDriver.ai"
+sidebarTitle: "Setup Guide"
+description: "Comprehensive guide to installing and setting up TestDriver.ai for your testing needs."
+---
+
+
+### **TestDriver.ai Installation Guide**
+
+Follow these steps to install and set up TestDriver.ai:
+
+---
+
+
+<Steps>
+
+  <Step title="Install NodeJS">
+
+    Ensure you have Node.js (v16 or higher) installed.
+
+    Check your Node.js version:
+    
+    ```bash
+    node -v
+    ```
+    
+    If not installed, download it from [Node.js Official Website](https://nodejs.org/).
+
+  </Step>
+  <Step title="Install NPM">
+    **npm**: Comes with Node.js. Verify `npm` is installed:
+    
+    ```bash
+    npm -v
+    ```
+  </Step>
+  <Step title="Install TestDriver">
+  
+    Install the TestDriver CLI globally using npm:
+  
+    ```bash
+    npm install -g testdriverai
+    ```
+  
+    Verify the installation:
+  
+    ```bash
+    testdriverai --version
+    ```
+  </Step>
+
+
+  <Step title="Set Up Your Test Runner">
+
+    TestDriver.ai requires a runner to execute tests. You can use either a hosted sandbox or your own computer.
+    
+    - **TestDriver Sandbox** - Run tests in a ephemeral linux virtual machine. Recommended for ease of use, added privacy, and parallelization. Requires a TestDriver API key.
+    - **Local Runner** - Run tests on your own computer. Recommended for working with existing user sessions, files, and applications. Requires additional setup.
+
+    <Tabs>
+      <Tab title="Hosted Sandbox Setup">
+
+        Run `testdriverai init` in your project directory and choose "yes" for the sandbox option. This will prompt you for your API key.
+
+        ```
+        Beginning setup...
+        ✔ Use TestDriver Sandbox Runners? (Recommended) … yes
+        ? API KEY (from https://app.testdriver.ai/team) › ********
+        ```
+
+        This will write the `TD_VM` and `TD_API_KEY` values to `.env` file in your project directory.
+
+      </Tab>
+      <Tab title="Local Windows Setup">
+
+        #### Install Python & Visual Studio Build Tools
+
+        ```powershell
+        choco install python visualstudio2022-workload-vctools -y
+        ```
+
+        #### Install NodeJS
+
+        You will also need NodeJS if you don't have it yet.
+
+        ```powershell
+        choco install nodejs-lts --version="20.17.0"
+        ```
+
+        #### Set Execution Policy
+
+        Open a new terminal with admin privileges and execute the following command : 
+        ```powershell
+        Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
+        ```
+      This gives TestDriver the right to execute it's scripts and is only valid for the current user. 
+      </Tab>
+      <Tab title="Local Mac Setup">
+        #### Enable Screen Recording Permissions
+        
+        TestDriver requires screen recording permissions to capture your screen during test execution.
+
+        - Open **System Preferences > Security & Privacy > Privacy > Screen Recording**.
+        - Check the box next to your terminal application (e.g., Terminal, iTerm2) to allow screen recording.
+        
+        This is required for TestDriver to capture your screen during test execution.
+        
+        #### Enable Accessibility Permissions
+        
+        - Open **System Preferences > Security & Privacy > Privacy > Accessibility**.
+        - Check the box next to your terminal application (e.g., Terminal, iTerm2) to allow accessibility features.
+
+        This is required for TestDriver to interact with your applications during test execution.
+
+      </Tab>
+    </Tabs>
+</Step>
+
+<Step title="Run TestDriver in Interactive Mode">
+  Start TestDriver in interactive mode. 
+  
+  ```bash
+  testdriverai
+  ```
+
+  You should see a prompt like this:
+
+  ```bash
+  Howdy! I'm TestDriver v4.2.26
+  ```
+</Step>
+
+</Steps>

package/docs/getting-started/writing.mdx

@@ -0,0 +1,99 @@
+---
+title: "Writing Tests with TestDriver.ai"
+sidebarTitle: "Writing Tests"
+description: "Step-by-step guide to writing and refining tests using TestDriver.ai's interactive CLI."
+---
+
+The TestDriver.ai interactive CLI allows you to create, refine, and execute tests dynamically using natural language commands. Here's how to get started.
+
+### Prepare Your Environment
+
+Before we get started, let's set up your machine to collaborate with TestDriver.
+
+#### Displays
+
+TestDriver isn't like any framework you've used before. TestDriver makes decisions based on what it can see on your display!
+
+<Tip>TestDriver only knows about what it can see on your primary display!</Tip>
+
+When you enter commands into TestDriver, the current window will minimize and the `focus-window` command will bring Chrome or other applications to the foreground. 
+
+For now, set up your environment with a browser window and your terminal side by side like so:
+
+![Terminal and Browser Side by Side](/images/content/side-by-side.webp)
+
+#### Application State
+
+The application you want to test should be visible before you run the testdriverai command. 
+
+For our example, make a new incognito window in Chrome and load our test webpage:
+
+```
+https://testdriverai.github.io/example-react-todo/
+```
+
+<Tip>Make sure to reset your application state before running every test!</Tip>
+
+### Launch Interactive Mode
+
+Run the following command to launch the TestDriver interactive mode:
+
+```bash
+testdriverai
+```
+You’ll see a prompt (`>`) where you can enter commands.
+
+### Write Tests Using Natural Language
+
+Simply describe what you want to test in plain English. For example:
+
+```
+> click sign up 
+```
+
+TestDriver will analyze your screen, interpret your instruction, and generate a test script to achieve the goal. It can see your UI, control the mouse and keyboard, and automate complex workflows—all through simple commands.
+
+```yaml
+thinking...
+
+To accomplish the goal of clicking "Sign Up," we need to
+focus on the Google Chrome application and then click on
+the "Sign Up" button.
+
+Here are the steps:
+
+1. Focus the Google Chrome application.
+2. Click on the "Sign Up" button.
+
+commands:
+  - command: focus-application
+    name: Google Chrome
+  - command: hover-text
+    text: Sign Up
+    description: button in the header
+    action: click
+
+command='focus-application' name='Google Chrome'
+command='hover-text' text='Sign Up' description='button in the header' action='click'
+```
+
+### 4. Refine Your Test
+
+If something doesn’t work as intended:
+
+- Use `/undo` to remove the last step:
+  ```
+  > /undo
+  ```
+
+Modify your prompt or add new steps to refine the test.
+
+### 5. Save Your Test
+
+Once you’re satisfied with the test, save it to a YAML file:
+
+```
+> /save
+```
+
+This generates a reusable YAML file (e.g., `testdriver.yaml`) with all the steps.

package/docs/guide/assertions.mdx

@@ -0,0 +1,195 @@
+---
+title: "Using Assertions in TestDriver.ai"
+sidebarTitle: "Assertions"
+description: "Comprehensive guide to understanding and implementing assertions in TestDriver.ai for robust test validation."
+---
+
+# Guide: Using Assertions in TestDriver.ai
+
+Assertions in TestDriver.ai allow you to validate that your application behaves as expected during a test. By using the `assert` command and visual assertions, you can ensure that specific conditions are met, such as verifying the presence of text, images, or UI elements on the screen.
+
+---
+
+## What Are Assertions?
+
+Assertions are checks that validate whether a specific condition is true. If the condition is not met, the test will fail, providing feedback on what went wrong.
+
+### Types of Assertions in TestDriver.ai:
+1. **Text Assertions**: Verify that specific text is visible on the screen.
+2. **Visual Assertions**: Validate the presence of images, icons, or UI elements.
+3. **Custom Assertions**: Use descriptive conditions to check for specific outcomes.
+
+---
+
+## How to Use the `assert` Command
+
+The `assert` command is used to validate conditions during a test. It checks whether the specified expectation is true.
+
+### Syntax:
+
+```yaml
+- command: assert
+  expect: <condition to check>
+  async: <true|false> # Optional, defaults to false
+
+```
+
+
+- **`expect`**: The condition to validate (e.g., "The login form is displayed").
+- **`async`**: (Optional) If set to `true`, the test will continue running without waiting for the assertion to pass.
+
+---
+
+### Example: Text Assertion
+
+#### YAML Command:
+
+```yaml
+- command: assert
+  expect: The login form is displayed
+
+```
+
+
+This assertion checks if the login form is visible on the screen.
+
+---
+
+### Example: Async Assertion
+
+#### YAML Command:
+
+```yaml
+- command: assert
+  expect: The success message is displayed
+  async: true
+
+```
+
+
+This assertion runs asynchronously, allowing the test to continue without waiting for the success message to appear.
+
+---
+
+## Visual Assertions
+
+Visual assertions validate the presence of images, icons, or UI elements on the screen. These are particularly useful for verifying non-text elements.
+
+### Example: Verifying an Image
+
+#### YAML Command:
+
+```yaml
+- command: hover-image
+  description: Company logo in the top-left corner
+  action: hover
+
+```
+
+
+This command hovers over the company logo to ensure it is present on the screen.
+
+---
+
+### Example: Verifying a Button
+
+#### YAML Command:
+
+```yaml
+- command: hover-text
+  text: Submit
+  description: Blue button with the text 'Submit' at the bottom of the form
+  action: hover
+
+```
+
+
+This command hovers over the "Submit" button to confirm its presence.
+
+---
+
+## Combining Assertions with Other Commands
+
+Assertions can be combined with navigation and interaction commands to validate workflows.
+
+### Example: Login Workflow with Assertions
+
+#### YAML Command:
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Open the homepage
+    commands:
+      - command: hover-text
+        text: Login
+        description: Login button in the top-right corner
+        action: click
+
+  - prompt: Verify the login form is displayed
+    commands:
+      - command: assert
+        expect: The login form is displayed
+
+  - prompt: Enter credentials and submit
+    commands:
+      - command: hover-text
+        text: Email
+        description: Email input field
+        action: click
+      - command: type
+        text: user@example.com
+      - command: hover-text
+        text: Password
+        description: Password input field
+        action: click
+      - command: type
+        text: password123
+      - command: hover-text
+        text: Submit
+        description: Submit button
+        action: click
+
+  - prompt: Verify the dashboard is displayed
+    commands:
+      - command: assert
+        expect: The dashboard is displayed
+
+```
+
+
+---
+
+## Debugging Assertions
+
+1. **Review Error Messages**:
+   - If an assertion fails, TestDriver.ai provides detailed error messages to help identify the issue.
+
+2. **Use Visual Feedback**:
+   - Leverage screenshots and visual feedback to verify the state of the application during the assertion.
+
+3. **Refine Descriptions**:
+   - Ensure that the `expect` condition or `description` is specific and matches the application's state.
+
+---
+
+## Best Practices for Assertions
+
+1. **Be Specific**:
+   - Use clear and concise conditions for assertions (e.g., "The login form is displayed").
+
+2. **Use Visual Assertions for Non-Text Elements**:
+   - Validate images, icons, and other UI elements using `hover-image` or `hover-text`.
+
+3. **Combine Assertions with Navigation**:
+   - Place assertions after navigation or interaction steps to validate the application's state.
+
+4. **Leverage Async Assertions**:
+   - Use `async: true` for non-blocking checks, especially for dynamic content.
+
+5. **Test Incrementally**:
+   - Add assertions step-by-step to validate each part of the workflow.
+
+---
+
+By using the `assert` command and visual assertions effectively, you can create robust and reliable tests that ensure your application behaves as expected.