@testdriverai/agent 7.8.0-test.38

package/docs/v6/features/visual-assertions.mdx

@@ -0,0 +1,139 @@
+---
+title: "Visual Assertions with TestDriver"
+sidebarTitle: "Visual Assertions"
+description: "Discover how TestDriver leverages AI-powered visual assertions to validate UI behavior and enhance end-to-end testing."
+icon: "binoculars"
+---
+
+Visual assertions in TestDriver allow you to validate that your application behaves as expected by analyzing what's displayed on the screen. This approach ensures that your tests mimic real user interactions and verify outcomes visually, making it ideal for testing UI-heavy applications and dynamic content.
+
+## What can TestDriver test with visual assertions?
+
+### AI chatbots
+
+- Validate that chatbot responses are displayed correctly.
+- Ensure the chatbot interface is accessible and functional.
+- See [AI Chatbots](/scenarios/ai-chatbot) for an example.
+
+### Desktop applications
+
+- Verify that desktop app windows, dialogs, and UI elements render as expected.
+- Test cross-platform compatibility for Windows, Mac, and Linux.
+- See [Using TestDriver with Desktop Apps](/apps/desktop-apps) for an example.
+
+### Chrome extensions
+
+- Confirm that the extension UI integrates properly with the browser.
+- Validate extension popups, settings, and interactions.
+- See [Using TestDriver with Chrome Extensions](/apps/chrome-extensions) for an example.
+
+### PDF generation
+
+- Assert that generated PDFs contain the correct text, images, and formatting.
+- Verify that PDFs open and display properly in viewers.
+- See [PDF Generation](/scenarios/pdf-generation) for an example.
+
+### Spelling & grammar
+
+- Check for spelling and grammar errors in displayed text.
+- Validate that text content matches expected language rules.
+- See [Spell Check](/scenarios/spell-check) for an example.
+
+### Auth signup & login
+
+- Ensure OAuth flows display the correct login screens.
+- Verify that users are redirected to the correct pages after login.
+- See [Login](/scenarios/log-in) for an example.
+
+### File system & uploads
+
+- Confirm that file upload dialogs appear and function correctly.
+- Validate that uploaded files are processed and displayed as expected.
+- See [File Upload](/scenarios/file-upload) for an example.
+
+### Image content
+
+- Verify that images are displayed correctly and match expected content.
+- Test for the presence of specific icons, logos, or visual elements.
+
+### Video content
+
+- Assert that videos play correctly and match expected visuals.
+- Validate video controls (play, pause, fullscreen) and captions.
+
+### OS accessibility features
+
+- Test high-contrast modes, screen readers, and other accessibility features.
+- Validate that UI elements are accessible and readable.
+
+### Light / Dark mode
+
+- Verify that the application switches between light and dark modes correctly.
+- Assert that UI elements are visible and styled appropriately in both modes.
+
+### Privacy configuration
+
+- Confirm that privacy settings are displayed and functional.
+- Validate that toggles, checkboxes, and options reflect the correct state.
+
+### `<iframe>`
+
+- Test content rendered inside `<iframe>` elements.
+- Validate interactions within embedded frames.
+
+### `<canvas>`
+
+- Verify that canvas elements render graphics, charts, or animations correctly.
+- Assert that dynamic content within the canvas matches expectations.
+
+### `<video>`
+
+- Ensure video elements load and play correctly.
+- Validate video overlays, subtitles, and playback controls.
+
+---
+
+## How to use visual assertions
+
+### Example: Asserting text on the screen
+
+```yaml
+- command: assert
+  expect: The chatbot response is displayed correctly
+```
+
+### Example: Asserting an image is present
+
+```yaml
+- command: assert
+  expect: The company logo appears in the header
+```
+
+### Example: Asserting video playback
+
+```yaml
+- command: assert
+  expect: The video is playing
+```
+
+---
+
+## Benefits of visual assertions
+
+1. **Selectorless Testing**: No need to rely on brittle selectors or static code.
+2. **Dynamic Adaptation**: Tests adapt to UI changes, reducing maintenance.
+3. **Real User Simulation**: Validates the application as a user would experience it.
+4. **Cross-Platform Support**: Works seamlessly across web, desktop, and mobile environments.
+
+---
+
+## Best Practices for visual assertions
+
+1. **Use Descriptive Prompts**: Provide clear descriptions for elements or behaviors you want to test.
+2. **Combine with Other Commands**: Use visual assertions alongside navigation and interaction commands for end-to-end testing.
+3. **Leverage AI Matching**: Use AI-powered matching for dynamic or non-deterministic content.
+4. **Test Across Modes**: Validate light/dark modes, accessibility features, and privacy settings for comprehensive coverage.
+
+---
+
+Visual assertions in TestDriver empower you to test a wide range of applications and scenarios with confidence, ensuring your software delivers a seamless user experience.

package/docs/v6/getting-started/ci.mdx

@@ -0,0 +1,146 @@
+---
+title: "Running in CI/CD"
+sidebarTitle: "Running in CI"
+description: "Learn how to set up and run tests with TestDriver in CI/CD pipelines using GitHub Actions as an example for seamless end-to-end testing."
+icon: "code-pull-request"
+---
+
+This guide explains how to set up TestDriver in CI/CD pipelines, using **GitHub Actions** as an example. The same principles can be applied to other CI/CD platforms like GitLab CI, Jenkins, or CircleCI.
+
+## Prerequisites
+
+1. **Add API key to secrets**: Obtain your API key from TestDriver and store it as a [CI/CD secret](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets) as `TD_API_KEY`.
+   Also make sure to add the `TD_WEBSITE` secret to the repository with the value of the website you are testing if its a web based application.
+2. **Test Files**: Ensure your test files are saved in the `testdriver/` directory of your repository.
+
+## Create a CI/CD workflow
+
+This example uses GitHub Actions, but the same approach can be adapted for other CI/CD platforms:
+
+1. Create a new file in your repository: `.github/workflows/testdriver.yaml`.
+
+<Note>
+  If you have already [initialized the project](/getting-started/setup) with
+  `npx testdriverai@latest init`, the workflow file will already be created.
+</Note>
+
+2. Add the following workflow configuration:
+
+### Example GitHub Actions workflow
+
+```yaml .github/workflows/testdriver.yaml [expandable]
+name: TestDriver / Run / Regressions
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 13 * * *" # Every day at 1 PM UTC (adjust if needed for timezone)
+
+permissions:
+  contents: read
+  pull-requests: read
+
+jobs:
+  test:
+    name: TestDriver
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Run TestDriver
+        run: npx testdriverai@latest run testdriver/testdriver.yaml
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          TD_WEBSITE: ${{ secrets.TD_WEBSITE }}
+```
+
+## Trigger the workflow
+
+1. Push changes to the `main` branch or open a pull request.
+2. Alternatively, manually trigger the workflow from the **Actions** tab in your GitHub repository.
+3. The scheduled trigger will run tests daily at 1 PM UTC for regression testing.
+
+## View results
+
+1. Navigate to the **Actions** tab in your GitHub repository (or equivalent CI/CD dashboard).
+2. Select the workflow run to view the test results. You will also find the link to the test recording at the end of the run, the url is of the format: `https://app.dashcam.io/replay/1234567890`
+
+<Tip>
+  You can also intregate our supported [reporting platform](/exporting/junit) to
+  get more currated results.
+</Tip>
+
+<Note>
+  For other CI/CD platforms, adapt the workflow syntax to match your platform's
+  requirements while keeping the same TestDriver CLI commands and environment
+  variables.
+</Note>
+
+# Debugging tests
+
+TestDriver provides a powerful debugging interface through its platform. This interface allows you to analyze test runs, identify failures, and optimize your test suite with detailed visual and textual feedback.
+
+### 1. Step-by-step execution logs
+
+- View each step of the test execution, including:
+  - The **action performed** (for example, clicking a button, typing text).
+  - The **expected outcome** (for example, verifying a specific element is visible).
+  - The **result** (pass, fail, or skipped).
+- Logs provide detailed context for each step, making it easier to pinpoint where and why a test failed.
+
+### 2. Visual feedback
+
+- **Screenshots**: See what the application looked like at each step of the test.
+- **GIF Previews**: Watch a replay of the entire test execution to understand the flow and identify UI issues.
+- **Highlighting**: Elements interacted with during the test are highlighted in screenshots, helping you verify that the correct elements were targeted.
+
+### 3. Error details
+
+- For failed steps, TestDriver provides:
+  - **Error messages**: Detailed descriptions of what went wrong (for example, "Element not found").
+  - **Stack traces**: For advanced debugging of backend or script-related issues.
+  - **Suggestions**: Recommendations for fixing common issues, such as adjusting prompts or improving element descriptions.
+
+### 4. Test history
+
+- Access the history of test runs to:
+  - Compare results across different builds or environments.
+  - Identify flaky tests by analyzing patterns in failures.
+  - Track improvements or regressions over time.
+
+### 5. Environment context
+
+- View the environment details for each test run, including:
+  - Operating system and browser version.
+  - Screen resolution and viewport size.
+  - Network conditions (if applicable).
+
+### 6. Collaboration tools
+
+- Share test results with your team by generating a shareable link.
+- Add comments or annotations to specific steps to facilitate discussions and debugging.
+
+---
+
+## How it works
+
+1. **Trigger**: The GitHub workflow is triggered based on the conditions defined in the `on` section.
+2. **Fetch the test files**: The test files are fetched from the repository based off the current branch.
+3. **Running the tests**: The TestDriver CLI is installed and the tests are run via the [`run`](/commands/run) command.
+4. **Authentication**: The `TD_API_KEY` environment variable authenticates your account.
+5. **Dashcam Recording**: Dashcam begins recording the test execution.
+6. **Test Execution**: The TestDriver CLI executes your commands on the cloud VM.
+7. **Test Summary**: TestDriver summarizes the test and sets the exit code based on the pass or fail state.
+8. **Cleanup**: The recording is uploaded to the TestDriver platform and the VM is destroyed, and all data is wiped.
+
+---
+
+## Additional features
+
+- **Parallel testing**: You can run multiple tests in [parallel](/features/parallel-testing#setting-up-parallel-testing-with-a-matrix-strategy). The number of parallel instances depends on the plan you are on.
+- **Environment Variables**: Pass [custom environment variables](guide/authentication#step-2%3A-pass-credentials-to-the-workflow) to configure credentials that are required for the tests.
+- **Chaining Workflows**: A common workflow involves waiting for staging to deploy and testing against the staging environment by passing in the deployed url as `TD_WEBSITE` or the staging build's installation url depending on the kind of application you are testing.

package/docs/v6/getting-started/cli.mdx

@@ -0,0 +1,91 @@
+---
+title: "Setup"
+sidebarTitle: "Setup"
+description: "Get started with TestDriver in minutes."
+icon: "download"
+mode: "wide"
+---
+
+<Steps>
+  <Step title="Set up your environment">
+      
+    Copy your API key from [the TestDriver dashboard](https://app.testdriver.ai/team), and set it as an environment variable.
+    
+    <Tabs>
+      <Tab title="macOS / Linux">
+        ```bash Export an environment variable on macOS or Linux systems
+        export TD_API_KEY="your_api_key_here"
+        ```
+      </Tab>
+      <Tab title="Windows">
+      ```powershell Export an environment variable in PowerShell
+      setx TD_API_KEY "your_api_key_here"
+      ```
+      </Tab>
+    </Tabs>
+
+  </Step>
+  <Step title="Try an example test">
+      
+    Download the TestDriver GitHub repository and run the web example test.
+
+    ```bash
+    git clone https://github.com/testdriverai/cli testdriverai
+    cd testdriverai/testdriver/examples/web
+    ```
+
+    TestDriver tests are written in YAML, a human-readable data format. The `example.yaml` file contains a series of steps that the agent will execute.
+
+    ```yaml testdriver/test.yaml
+    steps:
+      - prompt: log in
+      - prompt: add an item to the cart
+      - prompt: click on the cart icon
+      - prompt: complete checkout
+    ```
+
+    Each step has a `prompt` that describes what the agent should do. The agent will use the prompt to generate [commands](/commands/assert) that make the tests faster and more reliable the next time you run the test.
+
+  </Step>
+  <Step title="Generate test steps from prompts">
+    
+    Run the following command to run the test file. TestDriver will spawn a virtual machine, launch the sandbox test page, and execute the steps defined in the `example.yaml` file.
+
+    ```bash
+    npx testdriverai@latest run example.yaml --write --heal
+    ```
+
+    The `--write` flag tells TestDriver to save any generated commands to the test file, and the `--heal` flag allows TestDriver to recover from unexpected issues during the test run.
+
+  </Step>
+  <Step title="Run a regression test">
+
+    ```yaml
+      version: 6.0.0
+      steps:
+        - prompt: focus chrome
+          commands:
+            - command: focus-application
+              name: Google Chrome
+        - prompt: enter a username
+          commands:
+            - command: hover-text
+              text: Username
+              description: username input field
+              action: click
+            - command: type
+              text: standard_user
+        - prompt: enter a password
+          commands:
+            - command: hover-text
+              text: Password
+              description: password input field
+              action: click
+            - command: type
+              text: secret_password
+      ```
+
+      The `--write` command tells the agent to save the generated commands to the test file, and the `--heal` command gives the agent permission to recover if something goes wrong.
+
+  </Step>
+</Steps>

package/docs/v6/getting-started/editing.mdx

@@ -0,0 +1,100 @@
+---
+title: "Editing"
+sidebarTitle: "Editing"
+description: "Edit previously generated tests."
+icon: "file-pen"
+---
+
+Test scripts are written in YAML format, making them easy to read and modify manually if you need to.
+
+<Card title="Commands Reference" icon="link" href="/commands/assert">
+  Explore the available commands and their parameters.
+</Card>
+
+## Locate the Test File
+
+TestDriver test files are typically stored in the `testdriver/` directory of your project. The default file is `testdriver/testdriver.yaml`.
+
+### Open the Test File
+
+Use your preferred text editor or IDE (for example, Visual Studio Code) to open the YAML file.
+
+### YAML Structure
+
+Each test file consists of:
+
+- **Version**: Specifies the TestDriver version that was used to generate the test file. This is used to lock the version of TestDriver that is used to run the test.
+- **Session**: A unique identifier for the test session.
+- **Steps**: A list of prompts and commands to execute.
+
+```yaml test-file.yaml
+version: 6.0.0
+session: abc1234
+steps:
+  - prompt: Open Google Chrome and navigate to Airbnb
+    commands:
+      - command: focus-application
+        name: Google Chrome
+      - command: hover-text
+        text: Search Google or type a URL
+        description: main google search
+        action: click
+      - command: type
+        text: airbnb.com
+      - command: press-keys
+        keys:
+          - enter
+```
+
+<Tip>
+  The `version` captured when you generate a test file is the version of
+  TestDriver you used, and isn't used when running the test. This doesn't affect
+  your ability to use newer versions of TestDriver on existing test files nor do
+  these need to be updated when migrating to a new version.
+</Tip>
+
+### Modify the test steps
+
+You can add, edit, or remove steps as needed. Each step contains:
+
+- **Prompt**: A description of the action.
+- **Commands**: The specific actions to perform. It is an **optional** field, meaning, if there are no `commands` the AI considers the `prompt` and generates and executes the yaml on-the-go using the [exploratory mode](/interactive/explore)
+
+#### Example edits:
+
+- **Add a Wait Command**:
+
+  ```yaml
+  - command: wait
+    timeout: 3000
+  ```
+
+- **Change the URL**:
+  Update the `text` field in the [`type`](/commands/type) command:
+
+  ```yaml
+  - command: type
+    text: booking.com
+  ```
+
+### Validate the YAML syntax
+
+Ensure the YAML file is properly formatted:
+
+- Use consistent indentation (2 spaces recommended).
+- Avoid trailing spaces or tabs.
+- TestDriver will validate the yaml syntax and provide suggestions to fix any errors when you run the test.
+
+<Tip>
+  It is recommended to use our [VS Code extension](/getting-started/vscode)
+  while you are editing the yaml. It will provide you with a rich experience
+  with auto-completion, syntax highlighting, and validation.
+</Tip>
+
+### Test the changes
+
+Run the updated test to verify it works as expected:
+
+```bash
+npx testdriverai@latest run path/to/test.yaml
+```