testdriverai 5.2.2 → 5.3.0

package/docs/overview/faq.mdx

@@ -0,0 +1,122 @@
+---
+title: "FAQ"
+sidebarTitle: "FAQ"
+---
+
+
+## What is TestDriver?
+TestDriver is an AI-powered testing platform that simulates user interactions to automate end-to-end testing for web, desktop, and mobile applications.
+
+## How does TestDriver work?
+It interprets high-level prompts, interacts with interfaces like a user would, and verifies expected outcomes using visual assertions.
+
+## What platforms does TestDriver support?
+TestDriver supports Windows, Mac, Linux, desktop apps, chrome extensions, web browsers, and mobile interfaces (via emulator or device farm).
+
+## Can it be used for exploratory testing?
+Yes. TestDriver can autonomously navigate the application to generate new test cases.
+
+## Can it test desktop applications?
+Yes. It supports testing native desktop applications by simulating mouse and keyboard input and identifying UI elements.
+
+## Can it test mobile apps?
+Yes, via mobile emulators or integration with device farms.
+
+## Can TestDriver generate tests automatically?
+Yes, it explores the app and creates test cases based on UI flows and user interactions.
+
+## Can I create tests from natural language prompts?
+Yes. You can write high-level instructions in plain language, and TestDriver will interpret and build tests from them.
+
+## Can it generate tests from user stories or documentation?
+Yes. It can use minimal descriptions to produce complete test cases. 
+
+## Can it turn recorded user sessions into tests?
+No, an important part of AI-native testing is the intent behind the actions. TestDriver focuses on understanding user goals rather than just recording actions.
+
+## What happens when the UI changes?
+TestDriver adapts using AI—if a button or label changes, it can often infer the correct action without breaking.
+
+## Do I need to rewrite tests often?
+No. TestDriver reduces maintenance by handling common UI changes automatically.
+
+## How does it handle flaky tests?
+TestDriver Enterprise Dashboards provide insights into test stability, helping you identify flaky tests and their causes.
+
+## How are tests updated over time?
+TestDriver will open pull requests in your repository with updated tests when it detects changes in the UI or application behavior. You can also regenerate tests using the original prompts manually.
+
+## How does TestDriver report test failures?
+It provides detailed logs, screenshots, console output, and visual diffs.
+
+## What happens when a test fails?
+It stops execution, flags the failing step, and provides context for debugging.
+
+## Can I view why a test failed?
+Yes. You can view step-by-step logs, network traffic, DOM state, and video playback of the test run.
+
+## Can it automatically retry failed actions?
+Yes. You can configure retry behavior for individual steps or full tests.
+
+## Can I run tests in parallel?
+Yes. TestDriver supports parallel execution using multiple VMs or containers.
+
+## Can I track performance metrics during testing?
+Yes. It can log CPU, memory, load times, and frame rates to help catch performance regressions.
+
+## Can it validate non-deterministic output?
+Yes. It uses AI assertions to verify outcomes even when outputs vary (e.g., generated text or dynamic UIs).
+
+## Can it test workflows with variable inputs?
+Yes. It supports data-driven tests using parameterized inputs.
+
+## Can it test file uploads and downloads?
+Yes. TestDriver can interact with file pickers and validate uploaded/downloaded content.
+
+## Can it generate tests for PDFs or document output?
+Yes. It can open and verify generated files for expected text or formatting.
+
+## Can I trigger tests based on pull requests or merges?
+Yes. You can integrate TestDriver with your CI to trigger runs via GitHub Actions or other CI/CD tools.
+
+## Does it integrate with CI/CD tools?
+Yes. TestDriver integrates with pipelines like GitHub Actions, GitLab CI, and CircleCI.
+
+## Can I integrate TestDriver with Jira, Slack, etc.?
+Yes. You can receive alerts and sync test results with third-party tools via API/webhooks.
+
+## Does it support cloud and local environments?
+Yes. You can run tests locally or in the cloud using ephemeral VMs for clean state testing.
+
+## Does it work with existing test frameworks?
+It can complement or convert some existing test cases into its format, though full conversion depends on compatibility.
+
+## How does TestDriver measure test coverage?
+It tracks UI paths, element interaction frequency, and application state changes to infer coverage.
+
+## Can it suggest missing test scenarios?
+Yes. Based on interaction patterns and user behavior, it can propose additional test cases.
+
+## Can it analyze test stability over time?
+Yes. You can view trends in pass/fail rates and test execution consistency.
+
+## Is it safe to test sensitive data?
+Yes. TestDriver supports variable obfuscation, secure containers, and test data isolation.
+
+## Can I avoid using production data in tests?
+Yes. You can configure mock data, sanitize logs, and use test-specific environments.
+
+## How does the AI understand what to test?
+It uses language models to interpret your goals, element names, and interface cues to perform tasks.
+
+## Can I adjust how the AI interprets my prompt?
+Yes. You can rewrite prompts, add constraints, or review and tweak auto-generated steps.
+
+## Can I see what the AI is doing behind the scenes?
+Yes. You can inspect the resolved steps, see element matches, and adjust test flows before execution.
+
+## Can I use TestDriver to test new features?
+Yes. It’s great for validating changes, ensuring no regressions, and verifying rollout configurations.
+
+## Can it test seasonal or time-based behaviors?
+Yes. You can schedule tests or run them under specific date/time settings to verify time-sensitive logic.

package/docs/overview/quickstart.mdx

@@ -0,0 +1,66 @@
+---
+title: "Quickstart"
+---
+
+<Steps>
+
+<Step title="Install TestDriver">
+Install the TestDriver CLI globally:  
+```bash
+npm install -g testdriverai@beta
+```
+</Step>
+
+<Step title="Initialize Your Project">
+Set up your project with:  
+```bash
+testdriverai init
+```  
+This creates a `.env` file and sample workflow files.
+</Step>
+
+<Step title="Run TestDriver in Interactive Mode">
+Start TestDriver in interactive mode to create and refine tests dynamically:  
+```bash
+testdriverai
+```  
+Once started, you can use commands like `/try`, `/undo`, and `/save` to interactively build and test your flows.
+</Step>
+
+<Step title="Define Your Test Steps">
+Use natural language to describe what you want to test. For example:  
+```
+> Open Google Chrome and search for "testdriverai"
+```
+TestDriver will generate steps like:  
+```yaml
+- command: focus-application
+  name: Google Chrome
+- command: hover-text
+  text: Search Google or type a URL
+  description: main search bar
+  action: click
+- command: type
+  text: testdriverai
+- command: press-keys
+  keys: [enter]
+```
+</Step>
+
+<Step title="Refine and Save Your Test">
+- Use `/undo` to remove the last step if needed.  
+- Save your test with `/save` to generate a reusable YAML file.
+</Step>
+
+<Step title="Run Saved Tests">
+Once saved, you can run your test file anytime:  
+```bash
+testdriverai run path/to/test.yml
+```
+</Step>
+
+</Steps>
+
+---
+
+**Pro Tip:** Always start with interactive mode (`testdriverai`) to explore and refine your tests before saving them. It’s faster and more intuitive!

package/docs/overview/what-is-testdriver.mdx

@@ -0,0 +1,73 @@
+---
+title: "What is TestDriver?"
+description: "TestDriver is a computer-use agent for QA testing of user interfaces."
+---
+
+TestDriver uses AI vision and keyboard and mouse control to automate end-to-end testing. TestDriver is `selectorless` meaning it is not aware of the underlying code structure.
+
+- **Easier set up:** No need to craft complex selectors
+- **Less Maintenance:** Tests don't break when code changes
+- **More Power:** TestDriver can test any application and control any OS setting
+
+TestDriver is different from other computer-use agents in that it produces a `YAML` test script that increases the speed and repeatability of testing.
+
+### Selectorless Testing
+
+Unlike traditional frameworks (e.g., Selenium, Playwright), TestDriver doesn’t rely on CSS selectors or static analysis. Instead, tests are described in plain English, such as:
+
+```
+> Open Google Chrome and search for "testdriver"
+```
+
+This means that you can write tests without worrying about the underlying code structure:
+
+- Test any user flow on any website in any browser
+- Clone, build, and test any desktop app
+- Render multiple browser windows and popups like 3rd party auth
+- Test `<canvas>`, `<iframe>`,  and `<video>` tags with ease
+- Use file selectors to upload files to the browser
+- Resize the browser
+- Test chrome extensions
+- Test integrations between applications
+
+### The problem with current approach to end-to-end testing
+
+End-to-end is commonly described as the most expensive and time-consuming test method. Right now we write end-to-end tests using complex selectors that are tightly coupled with the code. 
+
+```js
+const e = await page.$('div[class="product-card"] >> text="Add to Cart" >> nth=2');
+```
+
+This tight coupling means developers need to spend time to understand the codebase and maintain the tests every time the code changes. And code is always changing!
+
+### End-to-end is about users, not code
+
+In end-to-end testing the business priority is usability. All that really matters is that the user can accomplish the goal. 
+
+TestDriver uses human language to define test requirements. Then our simulated software tester figures out how to accomplish those goals.
+
+Old and Busted (Selectors)  | New Hotness (TestDriver)
+|:----------------------|:-------------------|
+```div[class="product-card"] >> text="Add to Cart" >> nth=2``` | buy the 2nd product
+
+These high level instructions are easier to create and maintain because they are loosely coupled from the codebase. We're describing a high level goal, not a low level interaction.
+
+The tests will still continue to work even when the junior developer changes .product-card to .product.card or the designers change Add to Cart to Buy Now . The concepts remain the same so the AI will adapt. 
+
+## How exactly does this work?
+
+TestDriver uses a combination of reinforcement learning and computer vision. The context from successful text executions inform future executions. Here's an example of the context our model considers when locating a text match:
+
+Context| What is it? | Touchpoint
+|:----------|:------------------|:----------------|
+Prompt | Desired outcome | User Input
+Screenshot | Image of computer desktop | Runtime
+OCR | All possible text found on screen  Runtime
+Text Similarity | Closely matching text | Runtime 
+Redraw | Visual difference between previous and current desktop screenshots | Runtime
+Network | The current network activity compared to a baseline | Runtime
+Execution History | Previous test steps | Runtime
+System Information | Platform, Display Size, etc | Runtime
+Mouse Position | X, Y coordinates of mouse | Runtime
+Description | An elaborate description of the target element including it's position and function | Past Execution
+Text | The exact text value clicked | Past Execution

package/docs/quickstart.mdx

@@ -0,0 +1,66 @@
+---
+title: "Quickstart"
+---
+
+<Steps>
+
+<Step title="Install TestDriver">
+Install the TestDriver CLI globally:  
+```bash
+npm install -g testdriverai@beta
+```
+</Step>
+
+<Step title="Initialize Your Project">
+Set up your project with:  
+```bash
+testdriverai init
+```  
+This creates a `.env` file and sample workflow files.
+</Step>
+
+<Step title="Run TestDriver in Interactive Mode">
+Start TestDriver in interactive mode to create and refine tests dynamically:  
+```bash
+testdriverai
+```  
+Once started, you can use commands like `/try`, `/undo`, and `/save` to interactively build and test your flows.
+</Step>
+
+<Step title="Define Your Test Steps">
+Use natural language to describe what you want to test. For example:  
+```
+> Open Google Chrome and search for "testdriverai"
+```
+TestDriver will generate steps like:  
+```yaml
+- command: focus-application
+  name: Google Chrome
+- command: hover-text
+  text: Search Google or type a URL
+  description: main search bar
+  action: click
+- command: type
+  text: testdriverai
+- command: press-keys
+  keys: [enter]
+```
+</Step>
+
+<Step title="Refine and Save Your Test">
+- Use `/undo` to remove the last step if needed.  
+- Save your test with `/save` to generate a reusable YAML file.
+</Step>
+
+<Step title="Run Saved Tests">
+Once saved, you can run your test file anytime:  
+```bash
+testdriverai run path/to/test.yml
+```
+</Step>
+
+</Steps>
+
+---
+
+**Pro Tip:** Always start with interactive mode (`testdriverai`) to explore and refine your tests before saving them. It’s faster and more intuitive!

package/docs/security/action.mdx

@@ -0,0 +1,62 @@
+---
+title: "Security Features of the TestDriver Action"
+sidebarTitle: "GitHub Action"
+description: "Understand the ephemeral VM runners, secrets handling, and environment-specific security for the TestDriver Action."
+---
+
+# TestDriver Action Security
+
+## Open Source
+The TestDriver Action is open source, and its source code is available for review. You can find the repository here:
+
+[GitHub - testdriverai/action](https://github.com/testdriverai/action)
+
+## Ephemeral Virtual Machine Runners
+TestDriver tests are executed on private virtual machines (VMs) managed by Amazon EC2. These VMs are ephemeral, meaning they only exist for the duration of the test execution. Once the test is complete:
+- The VM is destroyed.
+- The hard disk is wiped to ensure no residual data remains.
+
+## Secrets
+To securely manage private information, we recommend storing sensitive data as secrets in your GitHub repository. Learn more about [storing secrets in GitHub](https://docs.github.com/en/actions/security-guides/encrypted-secrets).
+
+### Handling Secrets
+- **Prerun Scripts**: Any secrets supplied within prerun scripts or prompts are transmitted over SSL to the TestDriver API. Prerun scripts are **not persisted**.
+- **Agent Prompts**: Secrets supplied to agent prompts are persisted (see [Agent Security](../agent)).
+- **Secure Workflows**: If your workflow requires secret sharing and you encounter issues, please contact us for assistance.
+
+### Common Use Case
+A common workflow involves using prerun scripts to securely access a private staging website via basic authentication. This allows you to log into staging environments without persisting sensitive data on TestDriver servers.
+
+## Environment-Specific Security
+
+### Production
+Testing production environments is the simplest and most secure starting point. 
+- Production testing does not require any private information from your team.
+- Simply provide the tests to TestDriver and point them toward publicly available endpoints.
+- TestDriver does not need access to any private or sensitive information for production testing.
+
+### Staging
+Testing staging environments may require secure information, such as credentials or tokens. 
+- Use GitHub secrets to securely store and manage this information.
+- Refer to the **Secrets** section above for guidance on securely implementing tests for staging environments.
+
+### Development
+TestDriver can clone feature branches and build code on its virtual machines using workflows similar to GitHub Actions.
+
+#### GitHub Token for Development
+To test development branches of private codebases, you must supply a GitHub personal access token within the GitHub Action. This token is used to:
+- Clone the codebase onto the VM.
+- Build and test the code in an isolated environment.
+
+Example configuration:
+```yaml
+env:
+  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+```
+- The token is transmitted over SSL and is **not persisted**.
+- Learn more about managing the privacy of GitHub access tokens [here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).
+
+## Notes
+- TestDriver's ephemeral VMs ensure that no data persists beyond the test execution.
+- For maximum security, always use GitHub secrets to manage sensitive information.
+- If you have specific security concerns or requirements, feel free to contact us for support.

package/docs/security/agent.mdx

@@ -0,0 +1,62 @@
+---
+title: "Security Features of the TestDriver Agent"
+sidebarTitle: "TestDriver Agent "
+description: "Explore the data collected, persistence, and privacy practices of the TestDriver Agent."
+---
+
+# TestDriver Agent Security
+
+## Source
+The TestDriver Agent is open-source and available on NPM. You can browse the source code to see all the data collected and understand how everything works.
+
+## API
+The TestDriver Agent does not contain any AI models within it. Instead, it uploads desktop context to the TestDriver API, which uses that context to make decisions about what actions to perform. 
+
+The TestDriver API leverages OpenAI models behind the scenes. For more information about OpenAI and its privacy practices, visit their [privacy center](https://openai.com/privacy).
+
+## Desktop Context Collected
+During execution, the TestDriver Agent uploads the following information to the API:
+
+- **User Input Prompts**: The commands and prompts you provide to TestDriver.
+- **Active Window and Other Windows**: Information about the active window and other open windows, including application and window titles.
+- **System Information**: Details about the computer system running TestDriver.
+- **Mouse Position**: The current position of the mouse pointer.
+- **Desktop Screenshots**: Screenshots of the primary display.
+
+### Data Persistence
+- **Desktop Context**: All collected desktop context, except for screenshots, is persisted in the TestDriver database.
+- **Desktop Screenshots**: Screenshots are uploaded to the server for processing but are not stored in the database.
+
+### Desktop Screenshots
+TestDriver frequently captures screenshots of the desktop to provide the AI with decision-making context. These screenshots are uploaded to the API but are not persisted in the database. 
+
+- **Primary Display Only**: Screenshots are limited to the primary display.
+- **Privacy Recommendation**: For complete privacy, it is recommended to run TestDriver within a virtual machine on your desktop.
+
+**Important**: TestDriver cannot operate without visual context. Do not install TestDriver if you are uncomfortable with capturing images of the desktop.
+
+## Collected Data Details
+
+### Active Window
+Information about the open windows on the desktop is reported using the `active-window` module. This includes details such as the application name and window titles.
+
+### System Information
+System details, such as operating system, CPU, and memory, are reported using the `systeminformation` module.
+
+### User Prompts
+The prompts you input into TestDriver are uploaded to the API and stored in a database. This data is used to provide the AI with a history of context for better decision-making.
+
+## Additional Analytics
+When running `testdriver init`, you will be asked if you would like to share additional analytics. Sharing usage analytics is **opt-in**, and no extra data will be collected unless explicitly enabled.
+
+To disable additional analytics, set the `TD_ANALYTICS` environment variable to `false`:
+```bash
+TD_ANALYTICS=false
+```
+
+## Rate Limiting and Restrictions
+While the TestDriver Agent is free to use, TestDriver reserves the right to rate limit or restrict usage by IP address for any reason.
+
+## Notes
+- The TestDriver Agent is essential for providing the AI with the necessary context to operate effectively.
+- For privacy-conscious users, running TestDriver in a virtual machine is recommended to limit exposure of sensitive desktop information.

package/docs/security/platform.mdx

@@ -0,0 +1,54 @@
+---
+title: "Security Features of the Dashboard"
+sidebarTitle: "TestDriver Platform" 
+description: "Learn about the security features of the TestDriver web dashboard, including SSL, OAuth, RBAC, and more."
+---
+
+# TestDriver Web Dashboard Security
+
+## Overview
+The TestDriver web dashboard provides a secure interface for managing and reviewing your tests. Tests executed via the GitHub Action are recorded and reported through Dashcam, another application developed by TestDriver. For more details, refer to the [Dashcam documentation](https://dashcam.docs.testdriver.ai).
+
+Dashcam and TestDriver share the same API and web application back end, which includes robust privacy and security features.
+
+## Security Features
+
+### SSL
+- All data transmitted between your browser and the TestDriver web application is encrypted using HTTPS.
+
+### OAuth Authentication
+- Users can only authenticate via OAuth, provided by Auth0, ensuring secure and reliable user authentication.
+
+### Team Management
+- Administrators can add or remove individual team members.
+- Only administrators have the ability to manage team settings.
+
+### Role-Based Access Control (RBAC)
+- The first user to create a team is designated as the administrator.
+- Administrators:
+  - Are the only users who can view the API key.
+  - Can manage team settings.
+  - Cannot be removed from the team.
+- All other users are normal members with limited access.
+
+### API Key Rotation
+- Teams can rotate their API key for enhanced security.
+- It is recommended to rotate the API key every 90 days to minimize risk.
+
+### Secret Masking
+- Test replay logs and network requests are automatically scanned for sensitive information, such as:
+  - Credit card numbers
+  - Emails
+  - Passwords
+  - API keys
+- Detected secrets are masked with asterisks (`****`) to prevent exposure.
+
+### Encrypted At Rest
+- Test replays and logs are securely stored on Amazon S3 and encrypted at rest.
+- Test results are only accessible via temporary signed URLs.
+- Signed URLs are generated exclusively for team users and expire after a set duration.
+
+## Notes
+- The TestDriver web dashboard is designed with privacy and security as top priorities.
+- For additional security, ensure your team rotates API keys regularly and reviews team member access permissions.
+- If you have specific security concerns or questions, please contact TestDriver support.

package/docs/tutorials/advanced-test.mdx

@@ -0,0 +1,79 @@
+---
+title: "Tutorial: Advanced Test"
+description: "Explore advanced testing techniques and features in TestDriver."
+---
+
+# Advanced Test Tutorial
+
+Welcome to the Advanced Test Tutorial. In this guide, we will explore advanced testing techniques and features available in TestDriver. This tutorial assumes you have a basic understanding of TestDriver and its functionalities.
+
+## Prerequisites
+
+Before starting, ensure you have completed the basic TestDriver tutorial and have the following:
+
+- TestDriver installed and configured.
+- A sample project to test.
+
+## Advanced Features Overview
+
+TestDriver provides several advanced features to enhance your testing capabilities:
+
+1. **Custom Assertions**: Create custom assertions to validate specific conditions.
+2. **Parameterized Tests**: Run tests with different sets of input data.
+3. **Test Hooks**: Use hooks to execute code before or after tests.
+4. **Parallel Execution**: Speed up testing by running tests in parallel.
+
+## Step-by-Step Guide
+
+### Step 1: Setting Up Custom Assertions
+
+Custom assertions allow you to define specific conditions for your tests. Here's an example:
+
+```python
+def assert_custom_condition(value):
+    assert value > 0, "Value must be greater than 0"
+```
+
+### Step 2: Using Parameterized Tests
+
+Parameterized tests enable you to test multiple scenarios with different inputs. Example:
+
+```python
+import pytest
+
+@pytest.mark.parametrize("input,expected", [(1, True), (0, False)])
+def test_is_positive(input, expected):
+    assert (input > 0) == expected
+```
+
+### Step 3: Implementing Test Hooks
+
+Test hooks let you execute code before or after tests. Example:
+
+```python
+def setup_function():
+    print("Setting up test environment")
+
+def teardown_function():
+    print("Cleaning up test environment")
+```
+
+### Step 4: Running Tests in Parallel
+
+Parallel execution can be achieved using pytest-xdist. Install it with:
+
+```bash
+pip install pytest-xdist
+```
+
+Run tests in parallel using:
+
+```bash
+pytest -n 4
+```
+
+## Conclusion
+
+By leveraging these advanced features, you can create more robust and efficient tests for your projects. Experiment with these techniques to find the best fit for your testing needs.
+
+Happy testing!

package/docs/tutorials/basic-test.mdx

@@ -0,0 +1,41 @@
+---
+title: "Tutorial: Basic Test"
+description: "Learn how to create and run a basic test using TestDriver."
+---
+
+# Basic Test Tutorial
+
+This tutorial will guide you through the process of creating and running a basic test using TestDriver.
+
+## Prerequisites
+
+Before you begin, ensure you have the following:
+- TestDriver installed on your system.
+- Basic understanding of how TestDriver works.
+
+## Step 1: Create a Test
+
+To create a test, follow these steps:
+1. Open your code editor.
+2. Write the test script using TestDriver's syntax.
+3. Save the test script with a `.td` extension.
+
+Example:
+```plaintext
+test("Example Test", () => {
+  expect(1 + 1).toBe(2);
+});
+```
+
+## Step 2: Run the Test
+
+To run the test, use the following command:
+```plaintext
+testdriver run example-test.td
+```
+
+This will execute the test and display the results in the console.
+
+## Conclusion
+
+You have successfully created and run a basic test using TestDriver. Explore more advanced features to enhance your testing capabilities.

package/electron/overlay.html

@@ -16,6 +16,11 @@
       font-weight: 400;
       font-style: normal;
       font-size: 14px;
+      scrollbar-width: none; /* Hide scrollbars for Firefox */
+    }
+
+    *::-webkit-scrollbar {
+      display: none; /* Hide scrollbars for WebKit browsers */
     }
 
     @keyframes animate-glow {
@@ -189,10 +194,10 @@
     #terminal-wrapper {
       pointer-events: none;
       position: absolute;
-      left: calc(100vw - 700px);
+      right: calc(100vw - 600px);
       top: 0px;
       height: 100vh;
-      width: 700px;
+      width: 600px;
       background: black;
       opacity: 0;
       z-index: -1;
@@ -239,7 +244,6 @@
       <div id="mouse"><div id="dot"></div></div>
     </div>
     <div id="terminal-wrapper">
-      <img src="td.png" alt="td" style="position: absolute; top: 20; right: 20; height: 40px; z-index: 9999; background-color: black;">
       <div id="terminal"></div>
     </div>
   </div>