testdriverai 7.2.9 → 7.2.11

package/docs/v7/aws-setup.mdx

@@ -0,0 +1,338 @@
+---
+title: "AWS Setup Guide"
+sidebarTitle: "AWS Setup Guide"
+description: "Deploy TestDriver on your AWS infrastructure using CloudFormation"
+icon: "aws"
+---
+
+This guide walks you through setting up self-hosted TestDriver instances on AWS. By the end, you'll have a complete infrastructure that can spawn test environments on-demand and integrate with your CI/CD pipelines.
+
+```mermaid
+graph LR
+    A[Vitest] <--> B[api.testdriver.ai]
+    B <--> C[Your AWS EC2 Instance]
+```
+
+## Overview
+
+The setup process involves three main steps:
+
+1. **CloudFormation Infrastructure** — Deploy our template to create VPC, security groups, IAM roles, and launch templates
+2. **On-Demand Instance Spawning** — Use `spawn-runner.sh` to launch TestDriver instances when you need to run tests
+3. **CI/CD Integration** — Automate the lifecycle in GitHub Actions: spawn instance → run tests → terminate instance
+
+## Prerequisites
+
+Before you begin, ensure you have:
+
+- AWS account with CloudFormation permissions
+- [AWS CLI](https://aws.amazon.com/cli/) installed and configured (`aws configure`)
+- Access to the TestDriver AMI — [Contact us](https://form.typeform.com/to/UECf9rDx?typeform-source=testdriver.ai) with your AWS region
+- A GitHub repository for your tests
+
+<Tip>
+  The TestDriver Golden Image AMI ID is `ami-086b5b4b86d78987c`. Contact us to get access in your preferred AWS region.
+</Tip>
+
+## Step 1: Deploy CloudFormation Stack
+
+Our CloudFormation template creates all the AWS infrastructure you need:
+
+- Dedicated VPC with public subnet
+- Security group with required port access
+- IAM roles and instance profiles
+- EC2 launch template for instance creation
+
+Download the template from the [TestDriver CLI repository](https://github.com/testdriverai/testdriverai/tree/main/setup/aws/cloudformation.yaml), then deploy:
+
+```bash
+aws cloudformation deploy \
+  --template-file setup/aws/cloudformation.yaml \
+  --stack-name testdriver-infrastructure \
+  --parameter-overrides \
+    ProjectTag=testdriver \
+    AllowedIngressCidr=0.0.0.0/0 \
+    InstanceType=c5.xlarge \
+    CreateKeyPair=true \
+  --capabilities CAPABILITY_IAM
+```
+
+<Warning>
+  **Security**: Replace `AllowedIngressCidr=0.0.0.0/0` with your specific IP ranges to restrict VPC access.
+</Warning>
+
+### Get Your Launch Template ID
+
+After deployment completes, retrieve the launch template ID:
+
+```bash
+aws cloudformation describe-stacks \
+  --stack-name testdriver-infrastructure \
+  --query 'Stacks[0].Outputs[?OutputKey==`LaunchTemplateId`].OutputValue' \
+  --output text
+```
+
+<Tip>
+  **Save this ID** — you'll need it for spawning instances and CI configuration.
+</Tip>
+
+## Step 2: Spawn Test Instances
+
+Use the `spawn-runner.sh` script to launch instances on-demand. This script:
+
+- Launches a new EC2 instance using your launch template
+- Waits for the instance to be ready
+- Completes the TestDriver handshake
+- Returns instance details for test execution
+
+```bash
+# Set environment variables
+export AWS_REGION=us-east-2
+export AMI_ID=ami-086b5b4b86d78987c
+export AWS_LAUNCH_TEMPLATE_ID=lt-your-template-id
+export RESOLUTION=1440x900  # Optional, default is 1440x900
+
+# Spawn an instance
+./setup/aws/spawn-runner.sh
+```
+
+Output:
+```
+PUBLIC_IP=1.2.3.4
+INSTANCE_ID=i-1234567890abcdef0
+AWS_REGION=us-east-2
+```
+
+### Run Tests Against Your Instance
+
+With Vitest, specify the instance IP using the `TD_VM` environment variable:
+
+```bash
+TD_API_KEY=your-api-key TD_VM=1.2.3.4 npx vitest run
+```
+
+### Terminate Instances
+
+Always terminate instances after tests complete to avoid unnecessary costs:
+
+```bash
+aws ec2 terminate-instances \
+  --instance-ids i-1234567890abcdef0 \
+  --region us-east-2
+```
+
+<Note>
+  Instances are tagged with `Name=TestDriverRunner` and your `ProjectTag` for easy identification in the AWS console.
+</Note>
+
+## Step 3: GitHub Actions Integration
+
+Automate the full lifecycle in your CI/CD pipeline. This workflow spawns an instance, runs tests, and terminates the instance:
+
+```yaml .github/workflows/test.yml
+name: TestDriver Self-Hosted
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+# Prevent concurrent runs from competing for resources
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+
+      - name: Install dependencies
+        run: npm install
+
+      - name: Setup AWS Instance
+        id: aws-setup
+        run: |
+          OUTPUT=$(./setup/aws/spawn-runner.sh | tee /dev/stderr)
+          PUBLIC_IP=$(echo "$OUTPUT" | grep "PUBLIC_IP=" | cut -d'=' -f2)
+          INSTANCE_ID=$(echo "$OUTPUT" | grep "INSTANCE_ID=" | cut -d'=' -f2)
+          echo "public-ip=$PUBLIC_IP" >> $GITHUB_OUTPUT
+          echo "instance-id=$INSTANCE_ID" >> $GITHUB_OUTPUT
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          AWS_REGION: ${{ secrets.AWS_REGION }}
+          AWS_LAUNCH_TEMPLATE_ID: ${{ secrets.AWS_LAUNCH_TEMPLATE_ID }}
+          AMI_ID: ${{ secrets.AMI_ID }}
+
+      - name: Run Tests
+        run: npx vitest run
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          TD_VM: ${{ steps.aws-setup.outputs.public-ip }}
+
+      - name: Shutdown AWS Instance
+        if: always()
+        run: |
+          aws ec2 terminate-instances \
+            --region ${{ secrets.AWS_REGION }} \
+            --instance-ids ${{ steps.aws-setup.outputs.instance-id }}
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          AWS_REGION: ${{ secrets.AWS_REGION }}
+```
+
+### Required GitHub Secrets
+
+| Secret | Description | Example |
+|--------|-------------|---------|
+| `AWS_ACCESS_KEY_ID` | AWS access key | `AKIAIOSFODNN7EXAMPLE` |
+| `AWS_SECRET_ACCESS_KEY` | AWS secret key | `wJalrXUtnFEMI/K7MDENG...` |
+| `AWS_REGION` | AWS region | `us-east-2` |
+| `AWS_LAUNCH_TEMPLATE_ID` | From CloudFormation output | `lt-07c53ce8349b958d1` |
+| `AMI_ID` | TestDriver AMI ID | `ami-086b5b4b86d78987c` |
+| `TD_API_KEY` | Your TestDriver API key | From [console.testdriver.ai](https://console.testdriver.ai) |
+
+## AMI Customization
+
+The TestDriver Golden Image comes pre-configured with:
+
+- Windows Server with desktop environment
+- VNC + web server for remote access
+- Python, Node.js, Git
+- TestDriver agent and dependencies
+
+### Creating a Custom AMI
+
+You can customize the AMI to include additional software or configurations:
+
+<Steps>
+  <Step title="Connect via RDP">
+    Use the default credentials:
+    - **Username**: `testdriver`
+    - **Password**: `changemeABC123`
+  </Step>
+  
+  <Step title="Change the Password">
+    **Critical**: Run the password rotation script immediately:
+    ```powershell
+    C:\testdriver\RotateLocalPasswords.ps1
+    ```
+    Save the new password securely.
+  </Step>
+  
+  <Step title="Install Your Software">
+    Install any additional dependencies, configure settings, or modify the environment as needed.
+  </Step>
+  
+  <Step title="Create New AMI">
+    Use the AWS console or CLI to create an AMI from your modified instance. Update your workflow to use the new AMI ID.
+  </Step>
+</Steps>
+
+<Warning>
+  **Security**: Never use the default password in production. Always rotate passwords before creating custom AMIs.
+</Warning>
+
+### Changing Screen Resolution
+
+Set resolution via environment variable when spawning:
+
+```bash
+export RESOLUTION=1920x1080
+./setup/aws/spawn-runner.sh
+```
+
+Or in a lifecycle provision script:
+
+```yaml lifecycle/provision.yaml
+version: 7.0.0
+steps:
+  - prompt: set screen resolution
+    commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          C:\testdriver\SetResolution.ps1 -Width 1920 -Height 1080
+```
+
+## Security Best Practices
+
+### Network Security
+
+- **Restrict CIDR blocks** — Only allow access from known IP ranges
+- **Use VPC endpoints** — For private AWS service communication
+- **Enable VPC Flow Logs** — For network monitoring
+
+### AWS Authentication
+
+Use OIDC instead of long-term credentials for GitHub Actions:
+
+```yaml
+permissions:
+  id-token: write
+  contents: read
+
+steps:
+  - name: Configure AWS credentials
+    uses: aws-actions/configure-aws-credentials@v4
+    with:
+      role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
+      aws-region: us-east-2
+```
+
+See [GitHub's OIDC documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect) for setup instructions.
+
+### Instance Security
+
+- **Terminate immediately** after tests complete
+- **Monitor costs** with AWS billing alerts
+- **Use least-privilege IAM roles**
+- **Enable CloudTrail** for audit logging
+
+## Troubleshooting
+
+### Instance Not Responding
+
+- **Check security groups** — Verify required ports are open (RDP 3389, VNC 5900, TestDriver ports)
+- **Verify status checks** — Ensure instance has passed AWS status checks
+- **Check AMI compatibility** — Some instance types don't support certain AMIs
+
+### Connection Timeouts
+
+- Verify network connectivity from CI runner to instance
+- Check VPC routing and internet gateway configuration
+- Confirm instance is in the correct subnet
+
+### AWS CLI Errors
+
+- Validate credentials and permissions
+- Check service quotas and limits
+- Verify region consistency across all resources
+
+## Getting Help
+
+<CardGroup cols={2}>
+  <Card
+    title="Contact Support"
+    icon="envelope"
+    href="https://calendly.com/d/cq23-qyn-3v6/testdriver-ai-demo"
+  >
+    Get help with self-hosted setup and configuration
+  </Card>
+  <Card
+    title="Self-Hosted Overview"
+    icon="server"
+    href="/v7/self-hosted"
+  >
+    Learn more about self-hosted benefits and pricing
+  </Card>
+</CardGroup>

package/docs/v7/caching.mdx

@@ -0,0 +1,128 @@
+---
+title: "Caching Prompts"
+description: "1.7x faster test execution with intelligent caching and optimization"
+icon: "bolt"
+---
+
+TestDriver is engineered for performance with intelligent caching that delivers up to **1.7x faster** test execution by skipping redundant AI vision analysis.
+
+
+```javascript
+// First run: builds cache
+await testdriver.find('submit button');
+
+// Second run: exact match
+await testdriver.find('submit button');
+``` 
+
+## Automatic Caching
+
+Caching is enabled automatically with zero configuration. The cache key is computed from:
+
+- **File hash**: SHA-256 hash of the test file contents
+- **Selector prompt**: The exact text description passed to `find()`
+- **Screenshot context**: Perceptual hash of the current screen state
+- **Platform**: Operating system and browser version
+
+When you modify your test file, the hash changes automatically, invalidating stale cache entries and ensuring fresh AI analysis with your updated test logic.
+
+```javascript
+import { test } from 'vitest';
+import { chrome } from 'testdriverai/presets';
+
+test('auto-cached test', async (context) => {
+  const { testdriver } = await chrome(context, {
+    url: 'https://example.com'
+  });
+
+  // First call: AI analyzes screen, saves to cache
+  await testdriver.find('More information link'); // 2.1s
+
+  // Second call: cache hit, instant response
+  await testdriver.find('More information link'); // 12ms ⚡
+});
+```
+
+## Managing the Cache
+
+You can clear the cache within the TestDriver console. There, you'll also find previews of cached elements, the input prompts, as well as analytics on cache hit rates.
+
+<Card href="https://console.testdriver.ai/cache" title="TestDriver Cache" icon="database">
+  Manage and clear your test cache from the TestDriver console.
+</Card>
+
+## Debugging Cache Hits and Misses
+
+You can track cache performance in your tests:
+
+```javascript
+test('monitor cache performance', async (context) => {
+  const { testdriver } = await chrome(context, { url });
+
+  const element = await testdriver.find('submit button');
+
+  if (element.cacheHit) {
+    console.log('✅ Cache hit - instant response');
+    console.log('Strategy:', element.cacheStrategy); // 'exact', 'pixeldiff', or 'template'
+    console.log('Similarity:', `${(element.similarity * 100).toFixed(1)}%`);
+    console.log('Cache age:', element.cacheCreatedAt);
+  } else {
+    console.log('⏱️  Cache miss - AI analysis performed');
+    console.log('New cache entry created');
+  }
+});
+```
+
+## Configuring the Cache
+
+You can configure cache behavior globally when initializing TestDriver:
+
+```javascript
+import { TestDriver } from 'testdriverai';
+
+const testdriver = new TestDriver({
+  apiKey: process.env.TD_API_KEY,
+  cacheKey: 'my-test-suite', // cache-key for this instance
+  cacheDefaults: {
+    threshold: 0.05,      // 95% similarity
+  }
+});
+```
+
+It's also possible to override cache settings per `find()` call:
+
+```javascript
+// Default: 95% similarity required
+await testdriver.find('submit button');
+
+// Explicit strict threshold
+await testdriver.find('submit button', {
+  cacheThreshold: 0.01 // 99% similarity
+});
+```
+
+## Caching with Variables
+
+Custom cache keys prevent cache pollution when using variables in prompts, dramatically improving cache hit rates.
+
+```javascript
+// ❌ Without cache key - creates new cache for each variable value
+const email = 'user@example.com';
+await testdriver.find(`input for ${email}`); // Cache miss every time
+
+// ✅ With cache key - reuses cache regardless of variable
+const email = 'user@example.com';
+await testdriver.find(`input for ${email}`, {
+  cacheKey: 'email-input'
+});
+
+// Also useful for dynamic IDs, names, or other changing data
+const orderId = generateOrderId();
+await testdriver.find(`order ${orderId} status`, {
+  cacheKey: 'order-status'  // Same cache for all orders
+});
+```
+
+
+
+