testdriverai 5.2.1 → 5.3.0

package/docs/features/reusable-snippets.mdx

@@ -0,0 +1,124 @@
+---
+title: "Reusable Snippets in TestDriver.ai"
+sidebarTitle: "Reusable Snippets"
+description: "Discover how to modularize your test workflows using reusable YAML snippets in TestDriver.ai for improved maintainability and scalability."
+---
+
+Reusable snippets in TestDriver allow you to modularize your test steps by creating smaller, reusable YAML files that can be embedded into larger test workflows. This approach improves test maintainability, reduces duplication, and makes your test suite more organized and scalable.
+
+---
+
+## What Are Reusable Snippets?
+
+Reusable snippets are YAML files containing a set of test steps that perform a specific task, such as logging in, navigating to a page, or setting up test prerequisites. These snippets can be referenced in other test files using the `run` command, enabling you to reuse common actions across multiple tests.
+
+---
+
+## Why Use Reusable Snippets?
+
+1. **Maintainability**: Update a snippet in one place, and all tests using it will automatically reflect the changes.
+2. **Reusability**: Avoid duplicating common actions like login or setup across multiple tests.
+3. **Modularity**: Break down complex workflows into smaller, manageable pieces.
+4. **Consistency**: Ensure uniform behavior across tests by reusing the same logic.
+
+---
+
+## How to Create and Use Reusable Snippets
+
+### Step 1: Create a Snippet
+
+Save a YAML file containing the reusable steps. For example, create a `snippets/login.yaml` file for logging into an application:
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Log in to the application
+    commands:
+      - command: hover-text
+        text: Email address
+        description: email input field label
+        action: click
+      - command: type
+        text: ${TD_USERNAME} # Use environment variable for username
+      - command: hover-text
+        text: Password
+        description: password input field label
+        action: click
+      - command: type
+        text: ${TD_PASSWORD} # Use environment variable for password
+      - command: hover-text
+        text: Log In
+        description: log in button
+        action: click
+```
+
+---
+
+### Step 2: Reference the Snippet in a Test
+
+Use the `run` command to include the snippet in your main test file. For example:
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Log in and navigate to the dashboard
+    commands:
+      - command: run
+        file: snippets/login.yaml
+      - command: hover-text
+        text: Dashboard
+        description: dashboard link in the navigation bar
+        action: click
+```
+
+---
+
+### Step 3: Parameterize Inputs
+
+Use environment variables to pass dynamic data like usernames and passwords. Define these variables in your `.env` file or CI/CD pipeline:
+
+```bash
+TD_USERNAME=your_username
+TD_PASSWORD=your_password
+```
+---
+
+## Example: Combining Multiple Snippets
+
+You can chain multiple snippets together to create complex workflows. For example:
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Log in, search for a product, and add it to the cart
+    commands:
+      - command: run
+        file: snippets/login.yaml
+      - command: run
+        file: snippets/search_product.yaml
+      - command: run
+        file: snippets/add_to_cart.yaml
+```
+
+---
+
+## Best Practices for Reusable Snippets
+
+1. **Organize Snippets**: Store reusable snippets in a dedicated `snippets/` directory for better organization.
+2. **Use Descriptive Names**: Name snippet files clearly (e.g., `login.yaml`, `setup.yaml`) to indicate their purpose.
+3. **Test Snippets Independently**: Validate each snippet before integrating it into larger workflows.
+4. **Parameterize Inputs**: Use placeholders or environment variables for dynamic data.
+5. **Document Snippets**: Add comments or documentation to explain the purpose and usage of each snippet.
+
+---
+
+## Benefits of Reusable Snippets
+
+- **Efficiency**: Save time by reusing existing logic.
+- **Scalability**: Easily extend your test suite by combining snippets.
+- **Consistency**: Ensure uniform behavior across tests.
+- **Reduced Maintenance**: Centralize updates to common actions.
+
+---
+
+Reusable snippets are a cornerstone of efficient test design in TestDriver.ai. By modularizing your tests, you can create a scalable, maintainable, and reusable test suite that adapts to your application's evolving needs. Start by creating snippets for common actions like login or navigation, and expand your library as your test suite grows.

package/docs/features/selectorless.mdx

@@ -0,0 +1,62 @@
+---
+title: "Selectorless Testing with TestDriver.ai"
+sidebarTitle: "Selectorless Testing"
+description: "Selectorless testing approach simplifies end-to-end testing by using natural language and AI vision."
+---
+
+Selectorless testing eliminates the need for brittle selectors like CSS classes, IDs, or XPath. Instead, TestDriver uses natural language prompts and AI-powered vision to interact with applications as a user would. This makes tests more resilient to UI changes and reduces maintenance overhead.
+
+- Selectorless testing focuses on **what the user sees** rather than **how the UI is implemented**.
+- Tests are resilient to changes like text updates, class renaming, or minor layout adjustments.
+- By using natural language and AI vision, TestDriver simplifies test creation and maintenance.
+
+## What is Selectorless Testing?
+
+The following is an example of a TestDriver test.
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Click the "Sign Up" button
+    commands:
+      - command: hover-text
+        text: Sign Up
+        description: button in the header for user registration
+        action: click
+  - prompt: Assert the registration form is displayed
+    commands:
+      - command: assert
+        expect: The registration form is visible
+```
+
+This allows TestDriver locates the target for `hover-text` based on its context and description. The agent will search for elements: in the following order.
+
+- `text` - exact element to match
+- `description` - a description of the element given the exact text is not found, or there are multiple matches
+- `prompt` - a high level prompt used to regenerate the test if no match is found
+
+### What Happens When "Sign Up" Changes to "Register"?
+
+If the button text changes to "Register," TestDriver's AI vision will still locate the button based on its context and description. You don't need to update the test manually. 
+TestDriver will then update the test to reflect the new UI by modifing the `text` field. Then, it will open a new pull request with the changes.
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Click the "Register" button
+    commands:
+      - command: hover-text
+        text: Register
+        description: button in the header for user registration
+        action: click
+```
+## Why Selectorless Testing?
+
+Traditional testing frameworks rely on selectors tightly coupled to the codebase. 
+For example:
+
+```javascript
+const button = await page.$('button[class="sign-up-btn"]');
+```
+
+If the class name changes, the test will break, requiring updates to the test code. Selectorless testing avoids this by focusing on the intent of the interaction rather than the implementation details.

package/docs/features/visual-assertions.mdx

@@ -0,0 +1,123 @@
+---
+title: "Visual Assertions with TestDriver.ai"
+sidebarTitle: "Visual Assertions"
+description: "Discover how TestDriver.ai leverages AI-powered visual assertions to validate UI behavior and enhance end-to-end testing."
+---
+
+Visual assertions in TestDriver.ai allow you to validate that your application behaves as expected by analyzing what is displayed on the screen. This approach ensures that your tests mimic real user interactions and verify outcomes visually, making it ideal for testing UI-heavy applications and dynamic content.
+
+## What Are Visual Assertions?
+
+Visual assertions check the state of the screen to confirm that specific elements, content, or behaviors are present. Instead of relying on selectors or static code analysis, TestDriver uses AI-powered vision to interpret the screen, ensuring robust and adaptive testing.
+
+---
+
+## What Can TestDriver.ai Test with Visual Assertions?
+
+### AI Chatbots
+  - Validate that chatbot responses are displayed correctly.
+  - Ensure the chatbot interface is accessible and functional.
+
+### Desktop Applications
+  - Verify that desktop app windows, dialogs, and UI elements render as expected.
+  - Test cross-platform compatibility for Windows, Mac, and Linux.
+
+### Chrome Extensions
+  - Confirm that the extension UI integrates properly with the browser.
+  - Validate extension popups, settings, and interactions.
+
+### PDF Generation
+  - Assert that generated PDFs contain the correct text, images, and formatting.
+  - Verify that PDFs open and display properly in viewers.
+
+### Spelling & Grammar
+  - Check for spelling and grammar errors in displayed text.
+  - Validate that text content matches expected language rules.
+
+### Auth Signup & Login
+  - Ensure OAuth flows display the correct login screens.
+  - Verify that users are redirected to the correct pages after login.
+
+### File System & Uploads
+  - Confirm that file upload dialogs appear and function correctly.
+  - Validate that uploaded files are processed and displayed as expected.
+
+### Image Content
+  - Verify that images are displayed correctly and match expected content.
+  - Test for the presence of specific icons, logos, or visual elements.
+
+### Video Content
+  - Assert that videos play correctly and match expected visuals.
+  - Validate video controls (play, pause, fullscreen) and captions.
+
+### OS Accessibility Features
+  - Test high-contrast modes, screen readers, and other accessibility features.
+  - Validate that UI elements are accessible and readable.
+
+### Light / Dark Mode
+  - Verify that the application switches between light and dark modes correctly.
+  - Assert that UI elements are visible and styled appropriately in both modes.
+
+### Privacy Configuration
+  - Confirm that privacy settings are displayed and functional.
+  - Validate that toggles, checkboxes, and options reflect the correct state.
+
+### `<iframe>`
+  - Test content rendered inside `<iframe>` elements.
+  - Validate interactions within embedded frames.
+
+### `<canvas>`
+  - Verify that canvas elements render graphics, charts, or animations correctly.
+  - Assert that dynamic content within the canvas matches expectations.
+
+### `<video>`
+  - Ensure video elements load and play correctly.
+  - Validate video overlays, subtitles, and playback controls.
+
+---
+
+## How to Use Visual Assertions
+
+### Example: Asserting Text on the Screen
+
+```yaml
+- command: assert
+  expect: "The chatbot response is displayed correctly"
+
+```
+
+### Example: Asserting an Image is Present
+
+```yaml
+- command: assert
+  expect: The company logo appears in the header
+```
+
+### Example: Asserting Video Playback
+
+```yaml
+- command: assert
+  expect: "The video is playing"
+
+```
+---
+
+## Benefits of Visual Assertions
+
+1. **Selectorless Testing**: No need to rely on brittle selectors or static code.
+2. **Dynamic Adaptation**: Tests adapt to UI changes, reducing maintenance.
+3. **Real User Simulation**: Validates the application as a user would experience it.
+4. **Cross-Platform Support**: Works seamlessly across web, desktop, and mobile environments.
+
+---
+
+## Best Practices for Visual Assertions
+
+1. **Use Descriptive Prompts**: Provide clear descriptions for elements or behaviors you want to test.
+2. **Combine with Other Commands**: Use visual assertions alongside navigation and interaction commands for end-to-end testing.
+3. **Leverage AI Matching**: Use AI-powered matching for dynamic or non-deterministic content.
+4. **Test Across Modes**: Validate light/dark modes, accessibility features, and privacy settings for comprehensive coverage.
+
+---
+
+Visual assertions in TestDriver.ai empower you to test a wide range of applications and scenarios with confidence, ensuring your software delivers a seamless user experience.

package/docs/getting-started/ci.mdx

@@ -0,0 +1,196 @@
+---
+title: "Getting Started with TestDriver.ai"
+sidebarTitle: "Getting Started"
+description: "Learn how to set up and run tests with TestDriver.ai using GitHub Actions for seamless end-to-end testing."
+---
+
+This guide explains how to set up a basic GitHub Actions workflow to run tests using **TestDriver.ai**.
+
+---
+
+## Step 1: Prerequisites
+
+1. **TestDriver.ai API Key**: Obtain your API key from TestDriver.ai and store it as a GitHub secret (e.g., `TESTDRIVER_API_KEY`).
+2. **Test Files**: Ensure your test files are saved in the `testdriver/` directory of your repository.
+
+---
+
+## Step 2: Create a GitHub Actions Workflow
+
+1. Create a new file in your repository: `.github/workflows/testdriver.yml`.
+2. Add the following workflow configuration:
+
+### Example Workflow: `.github/workflows/testdriver.yml`
+
+```yaml
+name: Run Tests with TestDriver.ai
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  run-tests:
+    name: Run TestDriver.ai Tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v2
+
+      - name: Run TestDriver.ai
+        uses: testdriverai/action@main
+        with:
+          key: ${{ secrets.TESTDRIVER_API_KEY }}
+          prompt: |
+            1. Run all tests in the testdriver directory
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          FORCE_COLOR: "3"
+
+
+```
+
+
+---
+
+## Step 3: Add Your Test Files
+
+1. Save your test files in the `testdriver/` directory of your repository.
+2. Example test file: `testdriver/test.yml`:
+
+```yaml
+version: 4.2.18
+steps:
+  - prompt: Open the homepage
+  - prompt: Verify the login button is visible
+  - prompt: Click the login button
+  - prompt: Verify the login form is displayed
+
+
+```
+
+
+---
+
+## Step 4: Trigger the Workflow
+
+1. Push changes to the `main` branch or open a pull request.
+2. Alternatively, manually trigger the workflow from the **Actions** tab in your GitHub repository.
+
+---
+
+## Step 5: View Results
+
+1. Navigate to the **Actions** tab in your GitHub repository.
+2. Select the workflow run to view the test results.
+
+---
+
+## Example Output
+
+- ✅ **Run TestDriver.ai Tests**: All tests passed.
+- ❌ **Run TestDriver.ai Tests**: 1 test failed. View logs for details.
+
+---
+
+By following this guide, you can quickly set up and run tests using TestDriver.ai on GitHub Actions.
+
+# Debugging Tests in TestDriver.ai (app.testdriver.ai)
+
+TestDriver.ai provides a powerful debugging interface through its **app.testdriver.ai** platform (formerly Dashcam). This interface allows you to analyze test runs, identify failures, and optimize your test suite with detailed visual and textual feedback.
+
+---
+
+## Key Features of app.testdriver.ai for Debugging
+
+### 1. **Step-by-Step Execution Logs**
+- View each step of the test execution, including:
+  - The **action performed** (e.g., clicking a button, typing text).
+  - The **expected outcome** (e.g., verifying a specific element is visible).
+  - The **result** (pass, fail, or skipped).
+- Logs provide detailed context for each step, making it easier to pinpoint where and why a test failed.
+
+---
+
+### 2. **Visual Feedback**
+- **Screenshots**: See what the application looked like at each step of the test.
+- **GIF Previews**: Watch a replay of the entire test execution to understand the flow and identify UI issues.
+- **Highlighting**: Elements interacted with during the test are highlighted in screenshots, helping you verify that the correct elements were targeted.
+
+---
+
+### 3. **Error Details**
+- For failed steps, app.testdriver.ai provides:
+  - **Error messages**: Detailed descriptions of what went wrong (e.g., "Element not found").
+  - **Stack traces**: For advanced debugging of backend or script-related issues.
+  - **Suggestions**: Recommendations for fixing common issues, such as adjusting prompts or improving element descriptions.
+
+---
+
+### 4. **Test History**
+- Access the history of test runs to:
+  - Compare results across different builds or environments.
+  - Identify flaky tests by analyzing patterns in failures.
+  - Track improvements or regressions over time.
+
+---
+
+### 5. **Environment Context**
+- View the environment details for each test run, including:
+  - Operating system and browser version.
+  - Screen resolution and viewport size.
+  - Network conditions (if applicable).
+
+---
+
+### 6. **Collaboration Tools**
+- Share test results with your team by generating a shareable link.
+- Add comments or annotations to specific steps to facilitate discussions and debugging.
+
+---
+
+## How to Debug Tests in app.testdriver.ai
+
+1. **Access Test Results**:
+   - Navigate to [app.testdriver.ai](https://app.testdriver.ai).
+   - Log in with your TestDriver.ai account.
+   - Select the test run you want to debug from the dashboard.
+
+2. **Analyze the Test Run**:
+   - Review the **step-by-step logs** to identify where the test failed.
+   - Use the **screenshots** and **GIF previews** to visually verify the application's behavior.
+
+3. **Investigate Failures**:
+   - Click on the failed step to view detailed error messages and suggestions.
+   - Check the **highlighted elements** in the screenshots to ensure the correct elements were targeted.
+
+4. **Iterate and Fix**:
+   - Update your test prompts or application code based on the findings.
+   - Re-run the test to verify the fix.
+
+---
+
+## Best Practices for Debugging in app.testdriver.ai
+
+1. **Use Descriptive Prompts**:
+   - Ensure your test prompts are clear and specific to reduce ambiguity.
+   - Example: Instead of "Click the button," use "Click the 'Sign In' button in the header."
+
+2. **Leverage Visual Feedback**:
+   - Use screenshots and GIFs to quickly identify UI issues, such as misaligned elements or missing buttons.
+
+3. **Track Flaky Tests**:
+   - Use the test history to identify tests that fail intermittently and investigate potential causes (e.g., timing issues or dynamic content).
+
+4. **Collaborate with Your Team**:
+   - Share test results with developers and QA engineers to resolve issues faster.
+
+5. **Optimize Test Steps**:
+   - Combine or simplify test steps to make debugging easier and reduce test execution time.
+
+---
+
+By leveraging the debugging tools in **app.testdriver.ai**, you can efficiently analyze and resolve test failures, ensuring your application meets the highest quality standards.