testdriverai 5.2.2 → 5.3.0

package/docs/integrations/netlify.mdx

@@ -0,0 +1,98 @@
+---
+title: "Netlify"
+---
+
+# Testing Netlify Deployments with TestDriver.ai GitHub Action
+
+This guide explains how to integrate **TestDriver.ai** with **Netlify deployments** using the **GitHub Actions workflow**. By combining these tools, you can automatically test your Netlify preview deployments or production builds to ensure they meet your quality standards before merging or releasing.
+
+---
+
+## Workflow Overview
+
+1. **Trigger Netlify Deployment**: Use Netlify's GitHub integration to deploy your application on every pull request or push to the main branch.
+2. **Run Tests on the Deployment URL**: Use the TestDriver.ai GitHub Action to test the deployed application using the Netlify deployment URL.
+3. **Report Results**: View test results in the GitHub Actions dashboard or as comments on the pull request.
+
+---
+
+## Prerequisites
+
+1. **Netlify GitHub Integration**: Ensure your repository is connected to Netlify for automatic deployments.
+2. **TestDriver.ai API Key**: Store your API key as a GitHub secret (e.g., `TESTDRIVER_API_KEY`).
+3. **Netlify Deployment URL**: Use the `DEPLOY_URL` environment variable provided by Netlify to access the deployment.
+
+---
+
+## GitHub Actions Workflow
+
+Here’s a complete workflow to test Netlify deployments with TestDriver.ai:
+
+### Workflow File: `.github/workflows/test-netlify.yml````yaml
+name: Test Netlify Deployment with TestDriver.ai
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+jobs:
+  test-netlify:
+    name: Test Netlify Deployment
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v2
+
+      - name: Wait for Netlify Deployment
+        id: netlify
+        run: |
+          echo "Waiting for Netlify deployment..."
+          echo "Deployment URL: $DEPLOY_URL"
+          if [ -z "$DEPLOY_URL" ]; then
+            echo "Netlify deployment URL not found. Exiting."
+            exit 1
+          fi
+
+      - name: Run Tests with TestDriver.ai
+        uses: testdriverai/action@main
+        with:
+          key: ${{ secrets.TESTDRIVER_API_KEY }}
+          prompt: |
+            1. Open the deployment URL: $DEPLOY_URL
+            2. Verify the homepage loads correctly
+            3. Click the "Sign Up" button
+            4. Fill out the registration form
+            5. Submit the form and verify the success message
+        env:
+          DEPLOY_URL: $DEPLOY_URL
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          FORCE_COLOR: "3"
+
+```
+
+---
+
+## Workflow Steps Explained
+
+### 1. **Wait for Netlify Deployment**
+Netlify automatically sets the `DEPLOY_URL` environment variable for each deployment. This step ensures the deployment URL is available before running tests.```yaml
+- name: Wait for Netlify Deployment
+  id: netlify
+  run: |
+    echo "Waiting for Netlify deployment..."
+    echo "Deployment URL: $DEPLOY_URL"
+    if [ -z "$DEPLOY_URL" ]; then
+      echo "Netlify deployment URL not found. Exiting."
+      exit 1
+    fi
+
+```
+
+---
+
+### 2. **Run Tests with TestDriver.ai**
+The TestDriver.ai GitHub Action runs tests on the deployed application using the deployment URL. The `prompt` field specifies the test steps to execute.
+

package/docs/integrations/vercel.mdx

@@ -0,0 +1,177 @@
+---
+title: "Vercel"
+---
+
+# Testing Vercel Deployments with TestDriver.ai GitHub Action
+
+This guide explains how to integrate **TestDriver.ai** with **Vercel deployments** using the **GitHub Actions workflow**. By combining these tools, you can automatically test your Vercel preview deployments or production builds to ensure they meet your quality standards before merging or releasing.
+
+---
+
+## Workflow Overview
+
+1. **Trigger Vercel Deployment**: Use Vercel's GitHub integration to deploy your application on every pull request or push to the main branch.
+2. **Run Tests on the Deployment URL**: Use the TestDriver.ai GitHub Action to test the deployed application using the Vercel deployment URL.
+3. **Report Results**: View test results in the GitHub Actions dashboard or as comments on the pull request.
+
+---
+
+## Prerequisites
+
+1. **Vercel GitHub Integration**: Ensure your repository is connected to Vercel for automatic deployments.
+2. **TestDriver.ai API Key**: Store your API key as a GitHub secret (e.g., `TESTDRIVER_API_KEY`).
+3. **Vercel Deployment URL**: Use the `VERCEL_URL` environment variable provided by Vercel to access the deployment.
+
+---
+
+## GitHub Actions Workflow
+
+Here’s a complete workflow to test Vercel deployments with TestDriver.ai:
+
+### Workflow File: `.github/workflows/test-vercel.yml`
+
+```yaml
+name: Test Vercel Deployment with TestDriver.ai
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+jobs:
+  test-vercel:
+    name: Test Vercel Deployment
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v2
+
+      - name: Wait for Vercel Deployment
+        id: vercel
+        uses: amondnet/vercel-action@v20
+        with:
+          vercel-token: ${{ secrets.VERCEL_TOKEN }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          vercel-args: '--prod' # Optional: Use '--prod' for production builds
+        env:
+          VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
+          VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}
+
+      - name: Run Tests with TestDriver.ai
+        uses: testdriverai/action@main
+        with:
+          key: ${{ secrets.TESTDRIVER_API_KEY }}
+          prompt: |
+            1. Open the deployment URL: ${{ steps.vercel.outputs.url }}
+            2. Verify the homepage loads correctly
+            3. Click the "Sign Up" button
+            4. Fill out the registration form
+            5. Submit the form and verify the success message
+        env:
+          DEPLOYMENT_URL: ${{ steps.vercel.outputs.url }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          FORCE_COLOR: "3"
+
+
+```
+
+---
+
+## Workflow Steps Explained
+
+### 1. **Wait for Vercel Deployment**
+The `amondnet/vercel-action` waits for the Vercel deployment to complete and retrieves the deployment URL. This URL is stored in the `steps.vercel.outputs.url` variable.
+
+```yaml
+- name: Wait for Vercel Deployment
+  id: vercel
+  uses: amondnet/vercel-action@v20
+  with:
+    vercel-token: ${{ secrets.VERCEL_TOKEN }}
+    github-token: ${{ secrets.GITHUB_TOKEN }}
+    vercel-args: '--prod'
+  env:
+    VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
+    VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}
+
+
+```
+
+---
+
+### 2. **Run Tests with TestDriver.ai**
+The TestDriver.ai GitHub Action runs tests on the deployed application using the deployment URL. The `prompt` field specifies the test steps to execute.
+
+```yaml
+- name: Run Tests with TestDriver.ai
+  uses: testdriverai/action@main
+  with:
+    key: ${{ secrets.TESTDRIVER_API_KEY }}
+    prompt: |
+      1. Open the deployment URL: ${{ steps.vercel.outputs.url }}
+      2. Verify the homepage loads correctly
+      3. Click the "Sign Up" button
+      4. Fill out the registration form
+      5. Submit the form and verify the success message
+  env:
+    DEPLOYMENT_URL: ${{ steps.vercel.outputs.url }}
+    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    FORCE_COLOR: "3"
+
+
+```
+
+---
+
+## Example TestDriver.ai Prompt
+
+The `prompt` field in the TestDriver.ai action specifies the steps to test the Vercel deployment. For example:```yaml
+prompt: |
+  1. Open the deployment URL: ${{ steps.vercel.outputs.url }}
+  2. Verify the homepage loads correctly
+  3. Click the "Sign Up" button
+  4. Fill out the registration form
+  5. Submit the form and verify the success message
+
+
+```
+
+---
+
+## Secrets Configuration
+
+Add the following secrets to your GitHub repository:
+1. **`TESTDRIVER_API_KEY`**: Your TestDriver.ai API key.
+2. **`VERCEL_TOKEN`**: Your Vercel API token.
+3. **`VERCEL_ORG_ID`**: Your Vercel organization ID.
+4. **`VERCEL_PROJECT_ID`**: Your Vercel project ID.
+
+---
+
+## Benefits of This Workflow
+
+1. **Automated Deployment Testing**: Automatically test every Vercel deployment, including preview and production builds.
+2. **Early Issue Detection**: Catch issues in pull requests before merging.
+3. **Detailed Feedback**: View test results directly in the GitHub Actions dashboard.
+4. **Seamless Integration**: Combine Vercel's deployment capabilities with TestDriver.ai's testing power.
+
+---
+
+## Example Output
+
+### GitHub Actions Dashboard:
+- ✅ **Test Vercel Deployment**: All tests passed.
+- ❌ **Test Vercel Deployment**: 1 test failed. View logs for details.
+
+### TestDriver.ai Logs:
+- **Step 1**: Opened the deployment URL.
+- **Step 2**: Verified the homepage loaded correctly.
+- **Step 3**: Clicked the "Sign Up" button.
+- **Step 4**: Filled out the registration form.
+- **Step 5**: Failed to verify the success message.
+
+---
+
+By integrating TestDriver.ai with Vercel deployments, you can ensure that every deployment is thoroughly tested, reducing the risk of bugs reaching production and improving the overall quality of your application.

package/docs/interactive/assert.mdx

@@ -0,0 +1,51 @@
+---
+title: "/assert"
+---
+
+# Command: `/assert`
+
+## Description
+The `/assert` command ensures that a specific condition is true within your test. This is useful for verifying that tasks were completed successfully, just as a user would observe.
+
+## Usage
+```bash
+/assert <criteria>
+```
+
+## Behavior
+- The `/assert` command generates an assertion based on the specified criteria.
+- TestDriver.ai takes a screenshot and uses it to verify that the condition described in the `expect` field is true.
+- If the condition is not met, the test will fail and exit immediately.
+
+## Example Usage
+
+### Basic Assertion
+```bash
+/assert No error message is displayed
+```
+This generates the following command:
+```yaml
+- command: assert
+  expect: There is no error message
+```
+
+### Asynchronous Assertion
+To speed up tests, use `async: true` to allow the test to continue without waiting for the assertion to pass:
+```yaml
+- command: assert
+  expect: There is no error message
+  async: true
+```
+
+## Protips
+- Use assertions sparingly to avoid slowing down your tests.
+- Combine `async: true` with assertions to improve test performance while still validating critical conditions.
+- Ensure the `expect` field clearly describes the condition to avoid ambiguity.
+
+## Gotchas
+- If the condition in `expect` is not met, the test will fail and exit immediately.
+- Overusing assertions can make tests slower and harder to maintain.
+
+## Notes
+- The `/assert` command is ideal for validating key checkpoints in your test workflow.
+- Use this command to ensure that critical tasks, such as error-free execution or successful navigation, are completed as expected.

package/docs/interactive/generate.mdx

@@ -0,0 +1,41 @@
+---
+title: "/generate"
+---
+
+# Command: `/generate`
+
+## Description
+The `/generate` command is an experimental feature that instructs TestDriver.ai to create its own exploratory prompts. This command is used in the "Generate a Test Suite" demo and is designed to help automate the creation of exploratory test cases.
+
+## Usage
+```bash
+/generate
+```
+
+## Behavior
+- The `/generate` command analyzes the display and generates exploratory prompts for testing.
+- Each exploratory test is saved as a simple Markdown file containing a list of steps.
+- TestDriver.ai generates 10 Markdown files every time the `/generate` command is called.
+- The generated files are stored in the `./testdriver/generate/*.md` directory.
+
+## Example Output
+Here’s an example of a generated test file (`test-search-function.md`):
+```markdown
+1. Click on the search icon.
+2. Type "real-time chat" into the search bar.
+3. Assert that search results are relevant and displayed.
+```
+
+## Protips
+- Use `/generate` to quickly create exploratory tests for regression testing or feature validation.
+- Combine `/generate` with the GitHub action to generate and run regression tests in parallel.
+- Review the generated Markdown files and convert them into YAML test scripts for reuse.
+
+## Gotchas
+- The `/generate` command is experimental and may not always produce perfect test cases. Review and refine the generated tests as needed.
+- Ensure the `./testdriver/generate/` directory exists and has write permissions.
+
+## Notes
+- The `/generate` command is ideal for automating the creation of exploratory test cases and generating regression tests.
+- Generated tests can be merged into a regression test suite for continuous testing and validation.
+```

package/docs/interactive/run.mdx

@@ -0,0 +1,36 @@
+---
+title: "/run"
+---
+
+## Description
+The `/run` command is used to execute a test plan from a specified file. This command performs each step defined in the test plan and outputs the results.
+
+## Usage
+```bash
+/run <file>
+```
+
+## Example Usage
+```bash
+testdriverai
+> /run helloworld.yml
+```
+This command runs the `helloworld.yml` test plan, executing each command in the file sequentially.
+
+## Behavior
+- TestDriver will execute the test plan, performing each command as defined in the file.
+- If the test completes successfully, the program will exit with code `0`.
+- If any failures occur during the test, the program will output the errors and exit with code `1`.
+
+## Protips
+- Ensure the test file path is correct and accessible before running the command.
+- Use descriptive filenames for your test plans to make them easier to identify.
+- Combine `/run` with debugging tools to troubleshoot failing tests.
+
+## Gotchas
+- This command will exit the program upon execution, so ensure all necessary setup is complete before running it.
+- Any errors in the test plan (e.g., invalid commands or missing arguments) will cause the test to fail.
+
+## Notes
+- The `/run` command is ideal for executing pre-created test plans in an interactive session.
+- Use this command to validate and debug your test plans during development.

package/docs/interactive/save.mdx

@@ -0,0 +1,53 @@
+---
+title: "/save"
+---
+
+## Description
+The `/save` command saves the current state of the test script to a file. This command generates a YAML file containing the history of executed commands and tasks, allowing you to reuse or modify the test script later.
+
+## Usage
+```bash
+/save
+```
+
+## Example Usage
+```bash
+testdriverai
+> /save
+
+  saving...
+
+  Current test script:
+
+  version: 4.0.0
+  steps:
+    - prompt: navigate to fiber.google.com
+      commands:
+        - command: focus-application
+          name: Google Chrome
+        - command: hover-text
+          text: Search Google or type a URL
+          description: main google search
+          action: click
+        - command: type
+          text: fiber.google.com
+        - command: press-keys
+          keys:
+            - enter
+```
+
+## Behavior
+- The `/save` command generates a YAML file with the current test script, including all executed steps and commands.
+- The file can be used as a reusable test plan for future executions.
+
+## Protips
+- Use `/save` frequently during interactive sessions to preserve your progress and avoid losing work.
+- Combine `/save` with `/run` to quickly test and iterate on your scripts.
+
+## Gotchas
+- Ensure you have write permissions in the directory where the file will be saved.
+- The saved script reflects the current state of the session. Any unexecuted commands will not be included.
+
+## Notes
+- The `/save` command is ideal for creating reusable test scripts from interactive sessions.
+- Use this command to document and share your test workflows with your team.

package/docs/interactive/undo.mdx

@@ -0,0 +1,47 @@
+---
+title: "/undo"
+---
+
+## Description
+The `/undo` command removes the last generated command or step from the current test script. This is useful for quickly correcting mistakes or removing unintended actions during an interactive session.
+
+## Usage
+```bash
+/undo
+```
+
+## Behavior
+- The `/undo` command removes the most recently added command or step from the test script.
+- You can call `/undo` multiple times to remove multiple commands or steps.
+
+## Example Usage
+
+### Before `/undo`
+```yaml
+- step: 
+    - command: scroll-until-text
+      text: Add to cart
+- step: 
+    - command: hover-text
+      text: Add to cart
+      action: click
+```
+
+### After `/undo`
+```yaml
+- step: 
+    - command: scroll-until-text
+      text: Add to cart
+```
+
+## Protips
+- Use `/undo` immediately after generating a command to quickly fix mistakes or refine your test script.
+- Combine `/undo` with `/save` to iteratively build and refine your test scripts.
+
+## Gotchas
+- The `/undo` command only removes the last generated command or step. If you need to undo multiple actions, call `/undo` repeatedly.
+- Once a session is closed, you cannot undo previously saved commands.
+
+## Notes
+- The `/undo` command is ideal for interactive sessions where you are experimenting with test generation.
+- This command helps maintain a clean and accurate test script by allowing quick corrections.

package/docs/issues.mdx

@@ -0,0 +1,9 @@
+---
+title: "Known Issues"
+---
+
+# Image Description Matches
+
+# Text Substrings
+
+# 

package/docs/overview/comparison.mdx

@@ -0,0 +1,82 @@
+
+### Application Support
+
+TestDriver operates a full desktop environment, so it can run any application.
+
+| Application       | TestDriver | Playwright | Selenium |
+|:-------------------|:--------------|:-----------|:---------|
+| Web Apps          | ✅            | ✅         | ✅       |
+| Desktop Apps      | ✅            |          |        |
+| Chrome Extensions | ✅            |          |        |
+
+
+### Testing Features
+
+TestDriver is AI first.
+
+| Feature               | TestDriver | Playwright | Selenium |
+|:-----------------------|:--------------|:-----------|:---------|
+| Test Generation       | ✅            |          |        |
+| Adaptive Testing      | ✅            |          |        |
+| Visual Assertions     | ✅            |          |        |
+| Self Healing          | ✅            |          |        |
+| Application Switching | ✅            |          |        |
+| GitHub Actions        | ✅            | ✅         |        |
+| Team Dashboard        | ✅            |          |        |
+| Team Collaboration    | ✅            |          |        |
+
+### Test Coverage
+
+TestDriver has more coverage than selector-based frameworks.
+
+| Feature            | TestDriver | Playwright | Selenium |
+|:--------------------|:--------------|:-----------|:---------|
+| Browser Viewport   | ✅            | ✅         | ✅       |
+| Browser App        | ✅            |          |        |
+| Operating System   | ✅            |          |        |
+| PDFs               | ✅            |          |        |
+| File System        | ✅            |          |        |
+| Push Notifications | ✅            |          |        |
+| Image Content      | ✅            |          |        |
+| Video Content      | ✅            |          |        |
+| `<iframe>`         | ✅            |          |        |
+| `<canvas>`         | ✅            |          |        |
+| `<video>`          | ✅            |          |        |
+
+### Debugging Features
+
+Debugging features are powered by [Dashcam.io](https://dashcam.io).
+
+| Feature            | TestDriver | Playwright | Selenium |
+|:--------------------|:--------------|:-----------|:---------|
+| AI Summary         | ✅            |          |        |
+| Video Replay       | ✅            | ✅         |        |
+| Browser Logs       | ✅            | ✅         |        |
+| Desktop Logs       | ✅            |          |        |
+| Network Requests   | ✅            | ✅         |        |
+| Team Dashboard     | ✅            |          |        |
+| Team Collaboration | ✅            |          |        |
+
+### Web Browser Support
+
+TestDriver is browser agnostic and supports any version of any browser.
+
+| Feature  | TestDriver | Playwright | Selenium |
+|:----------|:--------------|:-----------|:---------|
+| Chrome   | ✅            | ✅         | ✅       |
+| Firefox  | ✅            | ✅         | ✅       |
+| Webkit   | ✅            | ✅         | ✅       |
+| IE       | ✅            |          | ✅       |
+| Edge     | ✅            | ✅         | ✅       |
+| Opera    | ✅            |          | ✅       |
+| Safari   | ✅            |          | ✅       |
+
+### Operating System Support
+
+TestDriver currently supports Mac and Windows!
+
+| Feature  | TestDriver | Playwright | Selenium |
+|:----------|:--------------|:-----------|:---------|
+| Windows  | ✅            | ✅         | ✅       |
+| Mac      | ✅            | ✅         | ✅       |
+| Linux    |             | ✅         | ✅       |