@testdriverai/agent 7.8.0-canary.10

package/docs/v6/features/auto-healing.mdx

@@ -0,0 +1,144 @@
+---
+title: "Self-Healing in TestDriver"
+sidebarTitle: "Self-Healing"
+description: "Learn how TestDriver's self-healing feature keeps your tests resilient and up-to-date with minimal manual effort."
+icon: "bandage"
+---
+
+Self-healing is a powerful feature in TestDriver that ensures your tests remain resilient even when the application under test changes.
+
+Use `--heal` to enable self-healing in your test runs. This feature allows TestDriver to automatically recover when minor changes occur, such as text updates or small UI adjustments, without requiring manual intervention.
+
+## Why use self-healing?
+
+In modern software development, applications frequently undergo changes, whether it's UI updates, text modifications, or layout adjustments. These changes can lead to test failures, requiring developers to manually update test scripts to reflect the new state of the application.
+
+This means that if a test fails due to a change in the application, TestDriver will attempt to recover by finding the updated element or text and retrying the action. If successful, it will update the test script accordingly.
+
+This feature is particularly useful in continuous integration (CI) environments, where tests need to adapt to frequent changes in the application.
+
+## How to enable self-healing
+
+To enable self-healing in TestDriver, simply add the `--heal` and `--write` flags when running your tests. The `--heal` flag instructs TestDriver to recover when test fail due to minor changes in the application, while `--write` saves newly generated steps to the test file.
+
+```bash
+npx testdriverai@latest run my-tests.yaml --heal --write
+```
+
+## How self-healing works
+
+When TestDriver steps fail, the AI will progressively fall back to attempt to complete the test.
+
+When running with `--heal` and `--write`, TestDriver will update the test files locally with any changes made during self-healing. You can then use GitHub Actions to commit these changes and create a pull request for review.
+
+## Benefits of self-healing
+
+- **Reduced Maintenance**: Eliminates the need for manual updates to tests when minor UI changes occur.
+- **Increased Test Resilience**: Ensures tests adapt to changes without breaking.
+- **Continuous Integration**: Keeps your test suite aligned with the latest application changes.
+- **Developer Collaboration**: Automatically opens a PR, allowing developers to review and approve updates.
+
+## How self-healing works
+
+1. **Test Execution**: TestDriver runs your test suite as usual.
+2. **Failure Detection**: If a test fails due to a change in the application (for example, text or UI updates), the AI identifies the failure point.
+3. **Recovery Attempt**: The AI uses its adaptive capabilities to locate the updated element or text and retries the action.
+4. **Test Update**: If the recovery is successful, TestDriver updates the test script with the new element or text.
+5. **Commit and PR**: GitHub Actions can detect the changes and automatically commit them and create a pull request for review.
+
+### Example: Button text changes from "Submit" to "Send"
+
+<Steps>
+<Step title="Initial Test">
+
+```yaml {2}
+- command: hover-text
+  text: Submit
+  description: Submit button in the form
+  action: click
+```
+
+</Step>
+<Step title="Application Update">
+  - The button text is changed from "Submit" to "Send" in the application.
+</Step>
+
+<Step title="Test Execution">
+  - TestDriver runs the test and fails to find the "Submit" button.
+</Step>
+
+<Step title="Self-Healing">
+  - The AI detects the failure and searches for a similar element based on the `description`.
+  - It identifies the "Send" button as the updated element and retries the action.
+</Step>
+<Step title="Test Update">
+  - The test script is updated to reflect the new button text:
+  
+  ```yaml {2}
+  - command: hover-text
+    text: Send
+    description: Submit button in the form
+    action: click
+  ```
+</Step>
+<Step title="Commit and PR Creation">
+   - GitHub Actions detects the changes and commits the updated test to a new branch and opens a PR:
+     - **Branch Name**: `auto-heal-update-submit-to-send`
+     - **PR Title**: `Auto-Healed Test: Updated "Submit" to "Send"`
+
+</Step>
+</Steps>
+## Example GitHub Action for self-healing
+
+Here's how you can configure a GitHub Action to enable self-healing and automatically create PRs for any test updates:
+
+```yaml
+name: TestDriver Self-Healing
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  test:
+    name: "Run Tests with Self-Healing"
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Run TestDriver with Auto-Healing
+        run: npx testdriverai@latest run testdriver/onboarding.yaml --heal --write
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+
+      - name: Commit auto-healed changes
+        run: |
+          git config --local user.email "action@github.com"
+          git config --local user.name "GitHub Action"
+          git add testdriver/
+          git diff --staged --quiet || git commit -m "Auto-healed test updates"
+
+      - name: Create Pull Request
+        uses: peter-evans/create-pull-request@v5
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          commit-message: "Auto-healed test updates"
+          title: "Self-Healed Test Updates"
+          branch: auto-heal-updates
+          base: main
+          body: |
+            This PR contains auto-healed test updates generated by TestDriver.
+
+            Please review the changes to ensure they are correct before merging.
+```
+
+---
+
+## Limitations of Self-Healing
+
+- **Major UI Overhauls**: Self-healing is best suited for minor changes (for example, text updates, small layout adjustments). Significant UI changes may still require manual intervention.
+- **Ambiguous Changes**: If multiple elements match the updated criteria, the AI may require additional context to make the correct decision.
+- **Step Deletion**: If UI or wizard pages are removed, the AI may not be able to recover. In such cases, manual updates are necessary.

package/docs/v6/features/generation.mdx

@@ -0,0 +1,116 @@
+---
+title: "Test Generation with TestDriver"
+sidebarTitle: "Test Generation"
+description: "Simplify test creation with AI-driven exploratory testing."
+icon: "wand-sparkles"
+---
+
+Writing tests can be a tedious and time-consuming task. TestDriver can generate tests just by exploring your app. This guide will show you how to generate tests using TestDriver.
+
+# Generate exploratory tests
+
+Launch `testdriverai` in interactive mode with the following command:
+
+```bash
+npx testdriverai@latest
+```
+
+TestDriver will boot up.
+
+```bash
+❯ testdriverai
+Howdy! I'm TestDriver v6.0.0
+```
+
+Ensure your website or app is visible on your test runner's display.
+
+Then, run the `generate` command.
+
+```bash
+/generate web 10
+```
+
+This will generate 10 tests within the `testdriver/generate` directory. You can change the number of tests generated by changing the number after `10`. The generated tests will be in the form of a `.yaml` file, and you can run them using your test runner of choice (local or cloud hosted).
+
+```bash
+> /generate web 5
+
+thinking...
+```
+
+Now your directory structure will look like this:
+
+```bash
+testdriver
+├── generate
+│   ├── test-error-user-login.yaml
+│   ├── test-locked-out-user-login.yaml
+│   ├── test-navigation-to-shop-page.yaml
+│   ├── test-problem-user-login.yaml
+│   └── test-standard-user-login.yaml
+```
+
+Notice that each test file has no commands\! It's only a list of prompts. This is similar to an exploratory test.
+
+```yaml
+version: 5.3.11
+steps:
+  - prompt: Fill in the "Username" field with "error_user".
+  - prompt: Fill in the "Password" field with "secret_password".
+  - prompt: Click the "Sign in" button.
+  - prompt: Assert that there are issues during checkout and an error is displayed.
+```
+
+# Generate regression tests from exploratory tests
+
+Now it's time to generate the regression test.
+
+Run the tests with the [`run`](/commands/run) command and use the `--write` parameter:
+
+```bash
+npx testdriverai@latest run testdriver/generate/test-error-user-login.yaml --write
+```
+
+When a test has no `commands`, TestDriver falls back to the `prompt`.
+
+```
+> Fill in the "Username" field with "error_user".
+No commands found, running exploratory
+```
+
+When using the `--write` command, TestDriver appends any generated commands to the test file.
+
+So as your test run, TestDriver will append successful commands to the YAML file and save it.
+
+<YmlWarning />
+
+```yaml
+version: 5.3.11
+steps:
+  - prompt: Focus the browser window.
+    commands:
+      - command: focus-application
+        name: Google Chrome
+      - command: hover-text
+        text: Username
+        description: username input label in the login form
+        action: click
+  - prompt: Fill in the "Password" field with "secret_password".
+    commands:
+      - command: hover-text
+        text: Password
+        description: password input label in the login form
+        action: click
+      - command: type
+        text: secret_password
+      - command: hover-text
+        text: Password
+        description: password input field in the login form
+        action: click
+```
+
+Now the next time you run the test, TestDriver will use the commands instead of the prompt\!
+
+```bash
+npx testdriverai@latest run testdriver/generate/test-error-user-login.yaml
+```

package/docs/v6/features/parallel-testing.mdx

@@ -0,0 +1,151 @@
+---
+title: "Parallel Testing with TestDriver"
+sidebarTitle: "Parallel Testing"
+description: "Learn how to leverage GitHub Actions' matrix strategy to run TestDriver tests in parallel, reducing execution time and improving scalability."
+icon: "server"
+---
+
+Parallel testing allows you to run multiple tests simultaneously, significantly reducing the time required to execute your test suite. By leveraging GitHub Actions' **matrix strategy**, you can dynamically distribute your tests across multiple jobs, ensuring efficient and scalable test execution.
+
+---
+
+## Why use parallel testing?
+
+1. **Faster Execution**: Run multiple tests at the same time to reduce overall test duration.
+2. **Scalability**: Easily scale your testing efforts as your test suite grows.
+3. **Dynamic Distribution**: Automatically distribute tests across jobs using GitHub's matrix strategy.
+4. **Cost Efficiency**: Optimize resource usage by running tests in parallel.
+5. **Reliability**: Avoid failures from going beyond your [plan's max instances](https://testdriver.ai/pricing).
+
+---
+
+## Setting Up parallel testing with a matrix strategy
+
+<Steps>
+<Step titel="Organize Your Test Files">
+
+Ensure your test files are stored in a directory (for example, `testdriver/acceptance/`) and follow a consistent naming convention (for example, `login.yaml`, `signup.yaml`, etc.).
+
+</Step>
+
+<Step title="Define the GitHub Action Workflow">
+
+Here's an example of a GitHub Action workflow that uses the matrix strategy to run tests in parallel:
+
+```yaml
+name: Parallel Testing with TestDriver
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  gather-test-files:
+    name: Gather Test Files
+    runs-on: ubuntu-latest
+    outputs:
+      test_files: ${{ steps.test_list.outputs.files }}
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Find all test files
+        id: test_list
+        run: |
+          FILES=$(ls ./testdriver/acceptance/*.yaml)
+          FILENAMES=$(basename -a $FILES)
+          FILES_JSON=$(echo "$FILENAMES" | jq -R -s -c 'split("\n")[:-1]')
+          echo "files=$FILES_JSON" >> $GITHUB_OUTPUT
+
+  run-tests:
+    name: Run Tests in Parallel
+    needs: gather-test-files
+    strategy:
+      matrix:
+        test: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
+      fail-fast: false
+      max-parallel: 8
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Run test in headless mode
+        run: npx testdriverai@latest run testdriver/acceptance/${{ matrix.test }}
+        env:
+          FORCE_COLOR: 3
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          TD_WEBSITE: https://testdriver-sandbox.vercel.app
+          TD_THIS_FILE: ${{ matrix.test }}
+```
+
+</Step>
+</Steps>
+---
+
+### Explanation of the workflow
+
+1. **`gather-test-files` Job**:
+
+   - Collects all test files in the `testdriver/acceptance/` directory.
+   - Uses `basename` to get just the filenames without the full path.
+   - Outputs the list of test files as a JSON array for the matrix strategy.
+
+2. **`run-tests` Job**:
+
+   - Uses the matrix strategy to dynamically create a job for each test file.
+   - Runs each test file in parallel using `npx testdriverai@latest` with the `--headless` flag.
+   - The `max-parallel` setting limits the number of tests that can run in parallel to `8` (the max instances for the [Medium Team Plan](https://testdriver.ai/pricing)).
+
+3. **Matrix Strategy**:
+
+   - The `matrix.test` variable represents each test filename.
+   - Each job runs a single test file from the `testdriver/acceptance/` directory.
+
+4. **Environment Variables**:
+
+   - `TD_API_KEY`: Your TestDriver API key for authentication.
+   - `TD_WEBSITE`: The target website URL for testing if its a web application.
+   - `TD_THIS_FILE`: The current test file being executed, this will also be the title of the test recording.
+
+5. **Fail-Fast**:
+
+   - Set to `false` to ensure all tests run even if one fails.
+
+---
+
+## Benefits of using the matrix strategy
+
+1. **Dynamic Test Distribution**: Automatically adapts to the number of test files.
+2. **Scalable**: Easily handles large test suites by distributing tests across multiple jobs.
+3. **Efficient Resource Usage**: Runs tests in parallel, reducing idle time.
+
+---
+
+## Example output
+
+When this workflow runs:
+
+- Each test file in `testdriver/acceptance/` (for example, `login.yaml`, `signup.yaml`) is executed in its own job.
+- The results of all tests are displayed in the GitHub Actions dashboard with clear matrix job names.
+- Failed tests can be easily identified and debugged individually.
+
+---
+
+## Best practices
+
+1. **Organize Test Files**: Use a consistent naming convention for test files to simplify management.
+2. **Monitor Test Results**: Review the GitHub Actions dashboard to identify and debug failing tests.
+3. **Optimize Test Files**: Ensure each test file is self-contained and doesn't depend on the execution of other tests.
+4. **Use Fail-Fast Judiciously**: Enable `fail-fast: true` only if you want to stop all tests when one fails.
+
+---
+
+## Conclusion
+
+Parallel testing with the GitHub Action matrix strategy is a powerful way to speed up your TestDriver test suite. By dynamically distributing tests across multiple jobs, you can ensure efficient execution and scalability, making it easier to maintain high-quality software.

package/docs/v6/features/reusable-snippets.mdx

@@ -0,0 +1,131 @@
+---
+title: "Reusable Snippets in TestDriver"
+sidebarTitle: "Reusable Snippets"
+description: "Discover how to modularize your test workflows using reusable YAML snippets in TestDriver for improved maintainability and scalability."
+icon: "repeat"
+---
+
+import GitignoreWarning from "/snippets/gitignore-warning.mdx";
+
+Reusable snippets in TestDriver allow you to modularize your test steps by creating smaller, reusable YAML files that can be embedded into larger test workflows. This approach improves test maintainability, reduces duplication, and makes your test suite more organized and scalable.
+
+---
+
+## What are reusable snippets?
+
+Reusable snippets are YAML files containing a set of test steps that perform a specific task, such as logging in, navigating to a page, or setting up test prerequisites. These snippets can be referenced in other test files using the [`run`](/commands/run) command, enabling you to reuse common actions across multiple tests.
+
+---
+
+## Why use reusable snippets?
+
+1. **Maintainability**: Update a snippet in one place, and all tests using it will automatically reflect the changes.
+2. **Reusability**: Avoid duplicating common actions like login or setup across multiple tests.
+3. **Modularity**: Break down complex workflows into smaller, manageable pieces.
+4. **Consistency**: Ensure uniform behavior across tests by reusing the same logic.
+
+---
+
+## How to create and use reusable snippets
+
+<Steps>
+<Step title="Create a Snippet">
+
+Save a YAML file containing the reusable steps. For example, create a `snippets/login.yaml` file for logging into an application:
+
+```yaml
+version: 6.0.0
+steps:
+  - prompt: Log in to the application
+    commands:
+      - command: hover-text
+        text: Email address
+        description: email input field label
+        action: click
+      - command: type
+        text: ${TD_USERNAME} # Use environment variable for username
+      - command: hover-text
+        text: Password
+        description: password input field label
+        action: click
+      - command: type
+        text: ${TD_PASSWORD} # Use environment variable for password
+      - command: hover-text
+        text: Log In
+        description: log in button
+        action: click
+```
+
+</Step>
+
+<Step title="Reference the Snippet in a Test">
+
+Use the [`run`](/commands/run) command to include the snippet in your main test file. For example:
+
+```yaml
+version: 6.0.0
+steps:
+  - prompt: Log in and navigate to the dashboard
+    commands:
+      - command: run
+        file: snippets/login.yaml
+      - command: hover-text
+        text: Dashboard
+        description: dashboard link in the navigation bar
+        action: click
+```
+
+</Step>
+
+<Step title="Parameterize Inputs">
+
+Use environment variables to pass dynamic data like usernames and passwords. Define these variables in your `.env` file or CI/CD pipeline:
+
+```bash
+TD_USERNAME=your_username
+TD_PASSWORD=your_password
+```
+
+<GitignoreWarning/>
+</Step>
+</Steps>
+
+## Example: Combining multiple snippets
+
+You can chain multiple snippets together to create complex workflows. For example:
+
+```yaml
+version: 6.0.0
+steps:
+  - prompt: Log in, search for a product, and add it to the cart
+    commands:
+      - command: run
+        file: snippets/login.yaml
+      - command: run
+        file: snippets/search_product.yaml
+      - command: run
+        file: snippets/add_to_cart.yaml
+```
+
+---
+
+## Best practices for reusable snippets
+
+1. **Organize Snippets**: Store reusable snippets in a dedicated `snippets/` directory for better organization.
+2. **Use Descriptive Names**: Name snippet files clearly (for example, `login.yaml`, `setup.yaml`) to indicate their purpose.
+3. **Test Snippets Independently**: Validate each snippet before integrating it into larger workflows.
+4. **Parameterize Inputs**: Use placeholders or environment variables for dynamic data.
+5. **Document Snippets**: Add comments or documentation to explain the purpose and usage of each snippet.
+
+---
+
+## Benefits of reusable snippets
+
+- **Efficiency**: Save time by reusing existing logic.
+- **Scalability**: Easily extend your test suite by combining snippets.
+- **Consistency**: Ensure uniform behavior across tests.
+- **Reduced Maintenance**: Centralize updates to common actions.
+
+---
+
+Reusable snippets are a cornerstone of efficient test design in TestDriver. By modularizing your tests, you can create a scalable, maintainable, and reusable test suite that adapts to your application's evolving needs. Start by creating snippets for common actions like login or navigation, and expand your library as your test suite grows.

package/docs/v6/features/selectorless.mdx

@@ -0,0 +1,80 @@
+---
+title: "Selectorless Testing"
+sidebarTitle: "Selectorless Testing"
+description: "Selectorless testing approach simplifies end-to-end testing by using natural language and AI vision."
+icon: "eye"
+---
+
+Selectorless testing eliminates the need for brittle selectors like CSS classes, IDs, or XPath.
+
+Instead, TestDriver uses natural language prompts and AI-powered vision to interact with applications as a user would. This makes tests more resilient to UI changes and reduces maintenance overhead.
+
+- Selectorless testing focuses on **what the user sees** rather than **how the UI is implemented**.
+- Tests are resilient to changes like text updates, class renaming, or minor layout adjustments.
+- By using natural language and AI vision, TestDriver simplifies test creation and maintenance.
+
+## What's selectorless testing?
+
+The following is an example of a TestDriver test.
+
+```yaml
+version: 6.0.0
+steps:
+  - prompt: Click the "Sign Up" button
+    commands:
+      - command: hover-text
+        text: Sign Up
+        description: button in the header for user registration
+        action: click
+  - prompt: Assert the registration form is displayed
+    commands:
+      - command: assert
+        expect: The registration form is visible
+```
+
+This allows TestDriver locates the target for [`hover-text`](/commands/hover-text) based on its context and description. The agent will search for elements: in the following order.
+
+- `text` - exact element to match
+- `description` - a description of the element given the exact text isn't found, or there are multiple matches
+- `prompt` - a high level prompt used to regenerate the test if no match is found
+
+### What happens when "Sign Up" changes to "Register"?
+
+If the button text changes to "Register," TestDriver's AI vision will still locate the button based on its context and description. You don't need to update the test manually.
+TestDriver will then update the test to reflect the new UI by modifying the `text` field. Then, it will open a new pull request with the changes.
+
+```yaml
+version: 6.0.0
+steps:
+  - prompt: Click the "Register" button
+    commands:
+      - command: hover-text
+        text: Register
+        description: button in the header for user registration
+        action: click
+```
+
+<Note>
+  Need to add `--heal` flag during the run command to enable auto-healing of the
+  tests. Refer the [Auto-Healing](/getting-started/running#auto-healing) section
+  to know more.
+</Note>
+
+## Why selectorless testing?
+
+Traditional testing frameworks rely on selectors tightly coupled to the codebase.
+For example:
+
+```javascript
+const button = await page.$('button[class="sign-up-btn"]');
+```
+
+<Warning>
+  When using legacy frameworks, if the class name changes, the test will break,
+  requiring updates to the test code!
+</Warning>{" "}
+
+<Check>
+  Selectorless testing avoids this by focusing on the intent of the interaction
+  rather than the implementation details.
+</Check>