@testdriverai/agent 7.8.0-test.38

package/docs/v6/getting-started/self-hosting.mdx

@@ -0,0 +1,408 @@
+---
+title: "Self-Hosting TestDriver"
+sidebarTitle: "Self-Hosting"
+description: "Complete guide to self-hosting TestDriver instances on AWS"
+icon: "server"
+---
+
+```mermaid
+graph LR
+    A[CLI] <--> B[api.testdriver.ai]
+    B <--> C[Your AWS EC2 Instance]
+```
+
+Self-hosting TestDriver allows you to run tests on your own infrastructure, giving you full control over the environment, security, and configurations. This guide walks you through setting up and managing self-hosted TestDriver instances using AWS.
+
+## Why self host?
+
+Self-hosting TestDriver gives you complete control over your test execution environment:
+
+- **Enhanced security**: Get complete control over ingress and egress rules.
+- **Complete customization**: Modify the TestDriver Golden Image (our pre-configured AMI) to include custom dependencies, software, and configurations at launch time.
+- **Powerful Infrastructure**: Run tests on bare metal infrastructure that support emulators and simulators.
+
+You'll use the [TestDriver CLI repository](https://github.com/testdriverai/cli) which contains all the infrastructure templates and setup scripts needed for self-hosting.
+
+## Overview
+
+By the end of this guide, you'll have a complete self-hosted testing infrastructure that can:
+
+- Spawn TestDriver instances on-demand in your AWS account
+- Run tests on your own AWS infrastructure with custom configurations
+- Integrate seamlessly with GitHub Actions CI/CD workflows
+- Automatically clean up resources after test completion
+
+The setup process involves three main steps:
+
+1. **CloudFormation Infrastructure**: Deploy our `setup/aws/cloudformation.yaml` template to create the foundational AWS resources (VPC, security groups, IAM roles, and a launch template for instance creation).
+2. **On-Demand Instance Spawning**: Use `setup/aws/spawn-runner.sh` with your Launch Template ID to programmatically spawn TestDriver instances whenever you need to run tests.
+3. **GitHub Actions Integration**: Use `.github/workflows/self-hosted.yml` as a template for running tests in CI. This workflow demonstrates the complete lifecycle: spawning an instance, running tests, and shutting down the instance to minimize costs.
+
+## Prerequisites
+
+- AWS account with permissions to run CloudFormation.
+- [AWS CLI](https://aws.amazon.com/cli/) installed locally.
+
+  <Tip>
+    Be sure to run `aws configure` with your credentials
+  </Tip>
+- Access to the TestDriver AMI (Golden Image is `ami-086b5b4b86d78987c`)\
+  [Contact us with your preferred AWS Region for access](http://testdriver.ai/demo).
+- A GitHub repository for committing your tests & workflow.
+
+## Step 1: Set Up AWS Infrastructure
+
+### Deploy CloudFormation Stack
+
+Our [`setup/aws/cloudformation.yaml`](https://github.com/testdriverai/cli/tree/main/setup/aws/cloudformation.yaml) template creates:
+
+- Dedicated VPC with public subnet
+- Security group with proper port access
+- IAM roles and instance profiles
+- EC2 launch template for programmatic instance creation
+
+This is a one-time setup used to generate a template ID for launching instances.
+
+```bash
+# Deploy the CloudFormation stack
+aws cloudformation deploy \
+  --template-file setup/aws/cloudformation.yaml \
+  --stack-name my-testdriver-infrastructure \
+  --parameter-overrides \
+    ProjectTag=testdriver \
+    AllowedIngressCidr=0.0.0.0/0 \
+    InstanceType=c5.xlarge \
+    CreateKeyPair=true \
+  --capabilities CAPABILITY_IAM
+```
+
+<Danger>
+  **Security**: Replace `AllowedIngressCidr=0.0.0.0/0` with your specific IP ranges to lock down access to your VPC.
+</Danger>
+
+### Get Launch Template ID
+
+After CloudFormation completes, find the launch template ID in the stack outputs:
+
+```bash
+aws cloudformation describe-stacks \
+  --stack-name my-testdriver-infrastructure \
+  --query 'Stacks[0].Outputs[?OutputKey==`LaunchTemplateId`].OutputValue' \
+  --output text
+```
+
+<Tip>
+  **Save this ID** – you'll need it for the next step.
+</Tip>
+
+## Step 2: Spawn a New TestDriver Runner
+
+This step is performed **every time you want to run tests**. The `spawn-runner.sh` script launches a new EC2 instance on-demand for test execution.
+
+### Using spawn-runner.sh
+
+Our [`setup/aws/spawn-runner.sh`](https://github.com/testdriverai/cli/tree/main/setup/aws/spawn-runner.sh) script:
+
+- Launches a new EC2 instance using your launch template from Step 1
+- Waits for the instance to become ready
+- Completes the TestDriver handshake
+- Returns instance details (IP, instance ID) for CLI usage
+
+The script accepts parameters as either environment variables or CLI arguments:
+
+```bash
+# Launch an instance using environment variables
+export AWS_REGION=us-east-2
+export AMI_ID=ami-••••••••••  # Your TestDriver AMI (contact us to get one)
+export AWS_LAUNCH_TEMPLATE_ID=lt-••••••••••  # From CloudFormation output from step 1
+export RESOLUTION=1440x900 # Change screen resolution if desired (default is 1440x900)
+
+/bin/bash ./setup/aws/spawn-runner.sh
+```
+
+The script outputs:
+
+```
+PUBLIC_IP=1.2.3.4
+INSTANCE_ID=i-1234567890abcdef0
+AWS_REGION=us-east-2
+```
+
+<Note>
+  **Instance Lifecycle**: Instances spawned by this script will continue running until you manually terminate them. They are automatically tagged with `Name=TestDriverRunner` and `Project=[your ProjectTag value]` for easy identification in the AWS console.
+</Note>
+
+#### Changing Resolution in Lifecycle Files
+
+You can also change the resolution before running tests by adding an [`exec`](/commands/exec) command in your `lifecycle/provision.yaml` file:
+
+```yaml lifecycle/provision.yaml
+version: 6.0.0
+steps:
+  - prompt: set screen resolution
+    commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          C:\testdriver\SetResolution.ps1 -Width 1920 -Height 1080
+```
+
+This approach is useful when you need different resolutions for different test scenarios. See the [Lifecycle Files documentation](/guide/lifecycle) for more information about provision scripts.
+
+### CLI Usage
+
+Once you have an instance IP, run tests directly:
+
+```bash
+# Basic test execution
+npx testdriverai@latest run test.yaml --ip=1.2.3.4
+```
+
+You can use the `PUBLIC_IP` to target the instance you just spawned via `./setup/aws/spawn-runner.sh`:
+
+```sh
+npx testdriverai@latest run testdriver/your-test.yaml \
+  --ip="$PUBLIC_IP" \
+```
+
+### Terminating Instances
+
+After your tests complete, terminate the instance to avoid unnecessary costs:
+
+```bash
+# Terminate the instance
+aws ec2 terminate-instances --instance-ids $INSTANCE_ID --region $AWS_REGION
+```
+
+You can also terminate instances manually through the AWS console by searching for instances tagged with `Name=TestDriverRunner`.
+
+## Step 3: GitHub Actions Integration
+
+This step shows you how to automate the entire test lifecycle in CI/CD.
+
+### Example Workflow
+
+Our [`.github/workflows/self-hosted.yml`](https://github.com/testdriverai/cli/tree/main/.github/workflows/self-hosted.yml) demonstrates the complete workflow: spawning an EC2 instance, running your tests, and shutting down the instance automatically to minimize costs.
+
+The workflow uses the GitHub secrets you configure (see below) to authenticate with AWS and spawn instances on-demand:
+
+```yaml
+name: TestDriver Self-Hosted
+
+on:
+  workflow_dispatch:
+  push:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup AWS Instance
+        id: aws-setup
+        run: |
+          OUTPUT=$(./setup/aws/spawn-runner.sh | tee /dev/stderr)
+          PUBLIC_IP=$(echo "$OUTPUT" | grep "PUBLIC_IP=" | cut -d'=' -f2)
+          INSTANCE_ID=$(echo "$OUTPUT" | grep "INSTANCE_ID=" | cut -d'=' -f2)
+          echo "public-ip=$PUBLIC_IP" >> $GITHUB_OUTPUT
+          echo "instance-id=$INSTANCE_ID" >> $GITHUB_OUTPUT
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          AWS_REGION: ${{ secrets.AWS_REGION }}
+          AWS_LAUNCH_TEMPLATE_ID: ${{ secrets.AWS_LAUNCH_TEMPLATE_ID }}
+          AMI_ID: ${{ secrets.AMI_ID }}
+
+      - name: Run TestDriver
+        run: |
+          npx testdriverai run your-test.yaml \
+            --ip="${{ steps.aws-setup.outputs.public-ip }}"
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+
+      - name: Shutdown AWS Instance
+        if: always()
+        run: |
+          aws ec2 terminate-instances \
+            --region ${{ secrets.AWS_REGION }}
+            --instance-ids ${{ steps.aws-setup.outputs.instance-id }}
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+		  AWS_REGION: ${{ secrets.AWS_REGION }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+```
+
+### Required Secrets
+
+Configure these secrets in your GitHub repository:
+
+| Secret                   | Description                         | Example                                                      |
+| ------------------------ | ----------------------------------- | ------------------------------------------------------------ |
+| `AWS_ACCESS_KEY_ID`      | AWS access key                      | `AKIAIOSFODNN7EXAMPLE`                                       |
+| `AWS_REGION`             | AWS Region                          | `us-east-2`                                                  |
+| `AWS_SECRET_ACCESS_KEY`  | AWS secret key                      | `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`                   |
+| `AWS_LAUNCH_TEMPLATE_ID` | Launch template from CloudFormation | `lt-07c53ce8349b958d1`                                       |
+| `AMI_ID`                 | TestDriver AMI ID                   | `ami-086b5b4b86d78987c`                                      |
+| `TD_API_KEY`             | TestDriver API key                  | Your API key from [the dashboard](https://app.testdriver.ai) |
+
+## AMI Customization
+
+### Using the Base AMI
+
+Our TestDriver Golden Image (AMI) comes pre-configured with everything you need to run tests:
+
+**Operating System & Environment:**
+
+- Windows Server with desktop environment
+- VNC + web server for remote desktop access through the browser
+
+**Development Tools:**
+
+- Python (with pip)
+- Node.js (with npm)
+- Git
+
+**Test Infrastructure:**
+
+- TestDriver agent and dependencies
+- Optimized settings for test execution
+- Pre-configured networking for TestDriver CLI communication
+
+### Modifying the AMI
+
+<Danger>
+  **Change Default Password First**: The TestDriver AMI includes a default password for initial access. You MUST change this password before saving a new AMI. Never use the default password in production environments.
+</Danger>
+
+You can customize the AMI for your specific needs. Follow these steps carefully:
+
+#### Step 1: Initial Access
+
+Connect to your instance via RDP using the default credentials:
+
+- **Username**: `testdriver`
+- **Default Password**: `changemeABC123`
+
+#### Step 2: Change the Password
+
+Before doing anything else, change the default password:
+
+1. Open PowerShell on the instance
+2. Run the password rotation script:
+
+   ```powershell
+   C:\testdriver\RotateLocalPasswords.ps1
+   ```
+3. **Save the new password securely** - you'll need it for all future access to this instance
+
+<Note>
+  The password rotation script will generate a secure password for the `testdriver` account. Make sure to save this password in a secure location (password manager, secrets vault, etc.) before proceeding.
+</Note>
+
+#### Step 3: Make Your Customizations
+
+Now that you've secured the instance with a new password:
+
+1. **Make your changes** using the `testdriver` account with your new password
+2. Install additional software as needed
+3. Configure settings specific to your environment
+
+<Tip>
+  **Need to make further customizations or debug issues?** RDP back into the instance using the `testdriver` account with the new password you set in Step 2. This gives you full desktop access to install software, modify configurations, or troubleshoot problems before saving your custom AMI.
+</Tip>
+
+#### Step 4: Save Your Custom AMI
+
+1. **Create a new AMI** from your modified instance using the AWS console or CLI
+2. **Update your workflow** to use the new AMI ID
+
+### Amazon Image Builder
+
+For automated AMI builds, use [Amazon EC2 Image Builder](https://aws.amazon.com/image-builder/):
+
+```yaml
+# Example Image Builder pipeline
+Components:
+  - Name: testdriver-base
+    Version: 1.0.0
+    Platform: Windows
+    Type: BUILD
+    Data: |
+      name: TestDriver Custom Setup
+      description: Custom TestDriver AMI with additional software
+      schemaVersion: 1.0
+      phases:
+        - name: build
+          steps:
+            - name: InstallSoftware
+              action: ExecutePowerShell
+              inputs:
+                commands:
+                  - "# Your custom installation commands here"
+```
+
+## Security Considerations
+
+### Network Security
+
+1. **Restrict CIDR blocks**: Only allow access from your known IP ranges
+2. **Use VPC endpoints**: For private communication with AWS services
+3. **Enable VPC Flow Logs**: For network monitoring and debugging
+
+### AWS Authentication
+
+Use [OIDC for GitHub Actions](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect) instead of long-term credentials:
+
+```yaml
+permissions:
+  id-token: write
+  contents: read
+
+steps:
+  - name: Configure AWS credentials
+    uses: aws-actions/configure-aws-credentials@v4
+    with:
+      role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
+      aws-region: us-east-2
+```
+
+### Instance Security
+
+- **Terminate instances** immediately after use
+- **Monitor costs** with AWS billing alerts
+- **Use least-privilege IAM roles** for instance profiles
+- **Enable CloudTrail** for audit logging
+
+## Troubleshooting
+
+### Common Issues
+
+**Instance not responding in TestDriver CLI:**
+
+When the CLI displays connection errors or timeouts, check:
+
+- **Security group rules**: The CloudFormation template configures all necessary ports (RDP 3389, VNC 5900, and TestDriver communication ports). Verify your security group hasn't been modified.
+- **Instance status checks**: Ensure the instance has passed both system and instance status checks in the AWS console.
+- **AMI compatibility**: Verify the AMI is compatible with your selected instance type (some instance types don't support certain AMIs).
+
+**Connection timeouts:**
+
+- Verify network connectivity from runner to instance
+- Check VPC routing and internet gateway configuration
+- Confirm instance is in correct subnet
+
+**AWS CLI errors:**
+
+- Validate AWS credentials and permissions
+- Check AWS service quotas and limits
+- Verify region consistency across all resources
+
+### Getting Help
+
+For enterprise customers:
+
+- Contact your account manager for AMI access issues
+- Use support channels for infrastructure questions
+- Check the TestDriver documentation for CLI usage

package/docs/v6/getting-started/vscode.mdx

@@ -0,0 +1,88 @@
+---
+title: "Install the TestDriver VS Code Extension"
+sidebarTitle: "VS Code"
+description: "Comprehensive guide to installing and setting up TestDriver for VS Code"
+icon: "file-code"
+---
+
+<iframe
+  className="w-full aspect-video rounded-xl"
+  src="https://www.youtube.com/embed/dell_2tI6nc"
+  title="YouTube video player"
+  frameBorder="0"
+  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
+  allowFullScreen
+></iframe>
+
+The TestDriver VS Code extension is a powerful tool that integrates TestDriver's AI-driven test generation capabilities directly into your development environment:
+
+- Get started with example scripts for different platforms
+- Get syntax highlighting and autocompletion for [TestDriver YAML](/getting-started/editing)
+- Chat with the TestDriver agent to generate tests based on your requirements
+- Save and manage your tests directly within your project
+- Run and debug tests directly from the IDE
+- View test results and logs within the IDE
+
+<Tip>
+  The TestDriver VS Code Extension is currently in beta. If you encounter any
+  issues or have feedback, please let us know [in
+  Discord](https://discord.com/invite/cWDFW8DzPm)
+</Tip>
+
+<Steps>
+  <Step title="Install the VS Code Extension">
+    
+    Click the button below to install the TestDriver extension for your preferred IDE. Then, follow the setup guide and chat with TestDriver to create your first test.
+
+    <Card
+      horizontal
+      title="VS Code"
+      arrow
+      href="vscode:extension/testdriver.testdriver"
+      icon="/images/content/extension/vscode.svg"
+    ></Card>
+
+    <Card
+      horizontal
+      title="Cursor"
+      arrow
+    href="cursor:extension/testdriver.testdriver"
+    icon="/images/content/extension/cursor.svg"
+
+    > </Card>
+
+    <Card
+      horizontal
+      title="Windsurf"
+      arrow
+      href="windsurf:extension/testdriver.testdriver"
+      icon="/images/content/extension/windsurf.svg"
+    ></Card>
+
+  </Step>
+  <Step title="Set Your TestDriver API Key">
+    
+    After installing the extension, you'll need to set your TestDriver API key in the extension settings.
+
+    1. Open the command palette (Cmd+Shift+P or Ctrl+Shift+P).
+    2. Search for "Preferences: Open Settings (UI)" and select it.
+    3. In the search bar, type "TestDriver" to filter the settings.
+    4. Find the "TestDriver: Set API Key" field and enter your API key from [the TestDriver dashboard](https://app.testdriver.ai/team).
+
+    <Tip>You can also set the `TD_API_KEY` environment variable in your system if you prefer not to enter it directly in the extension settings.</Tip>
+
+  </Step>
+  <Step title="Choose a Platform">
+    Click on the TestDriver icon in the sidebar to open the extension panel and choose a platform to get started (Web, Desktop, or Mobile).
+  </Step>
+  <Step title="Create Your First Test">
+    Now chat with the TestDriver agent to create your first test! Describe what you want to test, and the agent will generate the test steps for you.
+  </Step>
+  <Step title="Run your test">
+    Run your test by clicking the "Run" button in the extension panel. You can view the test results and logs directly within VS Code's Test Explorer. [Learn more about running tests in VS Code](https://code.visualstudio.com/docs/debugtest/testing). 
+  </Step>
+</Steps>
+
+---
+
+**Pro Tip:** Always start with interactive mode (`testdriverai`) to explore and refine your tests before saving them. It's faster and more intuitive\!

package/docs/v6/guide/assertions.mdx

@@ -0,0 +1,189 @@
+---
+title: "Using Assertions in TestDriver"
+sidebarTitle: "Assertions"
+description: "Comprehensive guide to understanding and implementing assertions in TestDriver for robust test validation."
+icon: "square-check"
+---
+
+# Guide: Using assertions in TestDriver
+
+Assertions in TestDriver allow you to validate that your application behaves as expected during a test. By using the [`assert`](/commands/assert) command and visual assertions, you can ensure that specific conditions are met, such as verifying the presence of text, images, or UI elements on the screen.
+
+---
+
+## What are assertions?
+
+Assertions are checks that validate whether a specific condition is true. If the condition isn't met, the test will fail, providing feedback on what went wrong.
+
+### Types of assertions in TestDriver:
+
+1. **Text Assertions**: Verify that specific text is visible on the screen.
+2. **Visual Assertions**: Validate the presence of images, icons, or UI elements.
+3. **Custom Assertions**: Use descriptive conditions to check for specific outcomes.
+
+---
+
+## How to use the [`assert`](/commands/assert) command
+
+The [`assert`](/commands/assert) command is used to validate conditions during a test. It checks whether the specified expectation is true.
+
+### Syntax:
+
+```yaml
+- command: assert
+  expect: <condition to check>
+  async: <true|false> # Optional, defaults to false
+```
+
+- **`expect`**: The condition to validate (for example, "The login form is displayed").
+- **`async`**: (Optional) If set to `true`, the test will continue running without waiting for the assertion to pass.
+
+---
+
+### Example: Text assertion
+
+#### TestDriver command:
+
+```yaml
+- command: assert
+  expect: The login form is displayed
+```
+
+This assertion checks if the login form is visible on the screen.
+
+---
+
+### Example: Async assertion
+
+#### TestDriver command:
+
+```yaml
+- command: assert
+  expect: The success message is displayed
+  async: true
+```
+
+This assertion runs asynchronously, allowing the test to continue without waiting for the success message to appear.
+
+---
+
+## Visual assertions
+
+Visual assertions validate the presence of images, icons, or UI elements on the screen. These are particularly useful for verifying non-text elements.
+
+### Example: Verifying an image
+
+#### TestDriver command:
+
+```yaml
+- command: hover-image
+  description: Company logo in the top-left corner
+  action: hover
+```
+
+This command hovers over the company logo to ensure it's present on the screen.
+
+---
+
+### Example: Verifying a button
+
+#### TestDriver command:
+
+```yaml
+- command: hover-text
+  text: Submit
+  description: Blue button with the text 'Submit' at the bottom of the form
+  action: hover
+```
+
+This command hovers over the "Submit" button to confirm its presence.
+
+---
+
+## Combining assertions with other commands
+
+Assertions can be combined with navigation and interaction commands to validate workflows.
+
+### Example: Login workflow with assertions
+
+```yaml login.yaml highlight={12,13, 36, 37}
+version: 6.0.0
+steps:
+  - prompt: Open the homepage
+    commands:
+      - command: hover-text
+        text: Login
+        description: Login button in the top-right corner
+        action: click
+
+  - prompt: Verify the login form is displayed
+    commands:
+      - command: assert
+        expect: The login form is displayed
+
+  - prompt: Enter credentials and submit
+    commands:
+      - command: hover-text
+        text: Email
+        description: Email input field
+        action: click
+      - command: type
+        text: user@example.com
+      - command: hover-text
+        text: Password
+        description: Password input field
+        action: click
+      - command: type
+        text: password123
+      - command: hover-text
+        text: Submit
+        description: Submit button
+        action: click
+
+  - prompt: Verify the dashboard is displayed
+    commands:
+      - command: assert
+        expect: The dashboard is displayed
+```
+
+---
+
+## Debugging assertions
+
+1. **Review Error Messages**:
+
+   - If an assertion fails, TestDriver provides detailed error messages to help identify the issue.
+
+2. **Use Visual Feedback**:
+
+   - Leverage screenshots and visual feedback to verify the state of the application during the assertion.
+
+3. **Refine Descriptions**:
+   - Ensure that the `expect` condition or `description` is specific and matches the application's state.
+
+---
+
+## Best practices for assertions
+
+1. **Be Specific**:
+
+   - Use clear and concise conditions for assertions (for example, "The login form is displayed").
+
+2. **Use Visual Assertions for Non-Text Elements**:
+
+   - Validate images, icons, and other UI elements using [`hover-image`](/commands/hover-image) or [`hover-text`](/commands/hover-text).
+
+3. **Combine Assertions with Navigation**:
+
+   - Place assertions after navigation or interaction steps to validate the application's state.
+
+4. **Leverage Async Assertions**:
+
+   - Use `async: true` for non-blocking checks, especially for dynamic content.
+
+5. **Test Incrementally**:
+   - Add assertions step-by-step to validate each part of the workflow.
+
+---
+
+By using the [`assert`](/commands/assert) command and visual assertions effectively, you can create robust and reliable tests that ensure your application behaves as expected.