@testdriverai/agent 7.8.0-test.38

package/ai/skills/testdriver-aws-setup/SKILL.md

@@ -0,0 +1,448 @@
+---
+name: testdriver:aws-setup
+description: Deploy TestDriver on your AWS infrastructure using CloudFormation
+---
+<!-- Generated from aws-setup.mdx. DO NOT EDIT. -->
+
+This guide walks you through setting up self-hosted TestDriver instances on AWS. By the end, you'll have fully automated test infrastructure that spawns and terminates instances on-demand.
+
+```mermaid
+graph LR
+    A[Vitest Test] --> B[setup-aws hook]
+    B --> C[Spawns EC2]
+    C --> D[Runs Test]
+    D --> E[Terminates EC2]
+```
+
+## How It Works
+
+TestDriver automatically manages AWS EC2 instances for your tests:
+
+1. **Deploy CloudFormation** — One-time infrastructure setup
+2. **Configure Vitest** — Add one line to your config
+3. **Run Tests** — Instances spawn automatically, run tests, and terminate
+
+That's it! No manual instance management needed.
+
+# Quickstart
+
+<Steps>
+  <Step title="Deploy Infrastructure">
+    <Card 
+      title="Launch CloudFormation Stack" 
+      icon="aws" 
+      href="https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://v7-cloudformation-template.s3.us-east-2.amazonaws.com/cloudformation.yaml" 
+      horizontal 
+      arrow
+    >
+      One-click AWS setup
+    </Card>
+  </Step>
+  
+  <Step title="Add to Vitest Config">
+    ```javascript vitest.config.mjs
+    setupFiles: ['testdriverai/vitest/setup', 'testdriverai/vitest/setup-aws']
+    ```
+  </Step>
+  
+  <Step title="Run Tests">
+    ```bash
+    TD_OS=windows AWS_REGION=us-east-2 \
+      AWS_LAUNCH_TEMPLATE_ID=lt-xxx AMI_ID=ami-xxx \
+      vitest run
+    ```
+  </Step>
+</Steps>
+
+## Overview
+
+The setup process is simple:
+
+1. **Deploy CloudFormation** — Creates VPC, security groups, IAM roles, and launch templates
+2. **Configure Vitest** — Add `setup-aws` to automatically manage instance lifecycle
+3. **Run Tests** — Set `TD_OS=windows` with AWS credentials and instances spawn/terminate automatically
+
+
+## Prerequisites
+
+Before you begin, ensure you have:
+
+- AWS account with CloudFormation permissions
+- [AWS CLI](https://aws.amazon.com/cli/) installed and configured (`aws configure`)
+- Access to the TestDriver AMI — [Contact us](http://testdriver.ai/demo) with your AWS region
+- A GitHub repository for your tests
+
+<Tip>
+  The TestDriver Golden Image AMI ID is `ami-0504bf50fad62f312`. Contact us to get access in your preferred AWS region.
+</Tip>
+
+## Step 1: Deploy CloudFormation Stack
+
+Our CloudFormation template creates all the AWS infrastructure you need:
+
+- Dedicated VPC with public subnet
+- Security group with required port access
+- IAM roles and instance profiles
+- EC2 launch template for instance creation
+
+<Tabs>
+  <Tab title="GUI">
+    Click the button below to launch the CloudFormation stack in your AWS Console:
+
+    <Card 
+      title="Launch Stack" 
+      icon="aws" 
+      href="https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://v7-cloudformation-template.s3.us-east-2.amazonaws.com/cloudformation.yaml" 
+      horizontal 
+      arrow
+    >
+      Deploy TestDriver infrastructure with one click
+    </Card>
+
+    Configure the stack parameters:
+    - **Stack name**: `testdriver-infrastructure` (or your preferred name)
+    - **ProjectTag**: `testdriver`
+    - **AllowedIngressCidr**: Your IP range (e.g., `203.0.113.0/24`)
+    - **InstanceType**: `c5.xlarge` (recommended)
+    - **CreateKeyPair**: `true`
+
+    <Warning>
+      **Security**: Replace `AllowedIngressCidr` with your specific IP ranges to restrict VPC access. Avoid using `0.0.0.0/0` in production.
+    </Warning>
+
+    ### Get Your Launch Template ID
+
+    After the stack creation completes, navigate to the **Outputs** tab to find your `LaunchTemplateId`:
+
+    ![Launch Template ID](/images/content/self-hosted/launchtemplateid.png)
+
+    <Tip>
+      **Save this ID** — you'll need it for spawning instances and CI configuration.
+    </Tip>
+  </Tab>
+  <Tab title="CLI">
+    Download the template from the [TestDriver CLI repository](https://github.com/testdriverai/testdriverai/blob/main/setup/aws/cloudformation.yaml), then deploy:
+
+    ```bash
+    aws cloudformation deploy \
+      --template-file setup/aws/cloudformation.yaml \
+      --stack-name testdriver-infrastructure \
+      --parameter-overrides \
+        ProjectTag=testdriver \
+        AllowedIngressCidr=0.0.0.0/0 \
+        InstanceType=c5.xlarge \
+        CreateKeyPair=true \
+      --capabilities CAPABILITY_IAM
+    ```
+
+    <Warning>
+      **Security**: Replace `AllowedIngressCidr=0.0.0.0/0` with your specific IP ranges to restrict VPC access.
+    </Warning>
+
+    ### Get Your Launch Template ID
+
+    After deployment completes, retrieve the launch template ID:
+
+    ```bash
+    aws cloudformation describe-stacks \
+      --stack-name testdriver-infrastructure \
+      --query 'Stacks[0].Outputs[?OutputKey==`LaunchTemplateId`].OutputValue' \
+      --output text
+    ```
+
+    <Tip>
+      **Save this ID** — you'll need it for spawning instances and CI configuration.
+    </Tip>
+  </Tab>
+</Tabs>
+
+## Step 2: Configure Vitest
+
+Add the AWS setup hook to your `vitest.config.mjs`:
+
+```javascript vitest.config.mjs
+import { defineConfig } from 'vitest/config';
+import { config } from 'dotenv';
+import TestDriver from 'testdriverai/vitest';
+
+config(); // Load .env file
+
+export default defineConfig({
+  test: {
+    testTimeout: 900000,
+    hookTimeout: 900000,
+    maxConcurrency: 3,
+    reporters: [
+      'default',
+      TestDriver(),
+      ['junit', { outputFile: 'test-report.junit.xml' }]
+    ],
+    setupFiles: ['testdriverai/vitest/setup', 'testdriverai/vitest/setup-aws'],
+  },
+});
+```
+
+<Note>
+  **That's it!** The `setup-aws` hook automatically spawns and terminates instances when `TD_OS=windows` is set. No manual instance management needed.
+</Note>
+
+## Step 3: Write Your Tests
+
+Tests should use `context.ip || process.env.TD_IP` for the IP configuration:
+
+```javascript
+import { describe, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("My Test", () => {
+  it("should run on self-hosted instance", async (context) => {
+    const testdriver = TestDriver(context, {
+      ip: context.ip || process.env.TD_IP,
+    });
+
+    await testdriver.provision.chrome({ url: "https://example.com" });
+    // ... your test steps
+  });
+});
+```
+
+<Note>
+  **How it works**: When `TD_OS=windows` with AWS credentials, `context.ip` is automatically set by the setup hook. When running without AWS setup (cloud-hosted), both are undefined and TestDriver uses the cloud. When `TD_IP` is provided manually, it takes precedence.
+</Note>
+
+## Step 4: Run Tests
+
+### Locally
+
+```bash
+TD_OS=windows \
+AWS_REGION=us-east-2 \
+AWS_LAUNCH_TEMPLATE_ID=lt-xxx \
+AMI_ID=ami-0504bf50fad62f312 \
+vitest run
+```
+
+<Note>
+  Each test gets its own fresh EC2 instance that's automatically terminated after completion.
+</Note>
+
+### In GitHub Actions
+
+Automate testing with self-hosted instances in your CI/CD pipeline. TestDriver automatically spawns a fresh instance for each test, runs the test, and terminates the instance.
+
+```yaml .github/workflows/test.yml
+name: TestDriver Self-Hosted Windows Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Run Windows tests with self-hosted instances
+        run: npx vitest run examples/*.test.mjs
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          TD_OS: windows
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          AWS_REGION: us-east-2
+          AWS_LAUNCH_TEMPLATE_ID: ${{ secrets.AWS_LAUNCH_TEMPLATE_ID }}
+          AMI_ID: ${{ secrets.AMI_ID }}
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-results
+          path: test-report.junit.xml
+```
+
+<Note>
+  **Automatic Instance Management**: Setting `TD_OS=windows` with AWS credentials enables automatic instance spawning. Each test gets its own fresh instance that's terminated after the test completes.
+</Note>
+
+### Required GitHub Secrets
+
+| Secret | Description | Example |
+|--------|-------------|---------|
+| `AWS_ACCESS_KEY_ID` | AWS access key | `AKIAIOSFODNN7EXAMPLE` |
+| `AWS_SECRET_ACCESS_KEY` | AWS secret key | `wJalrXUtnFEMI/K7MDENG...` |
+| `AWS_REGION` | AWS region | `us-east-2` |
+| `AWS_LAUNCH_TEMPLATE_ID` | From CloudFormation output | `lt-07c53ce8349b958d1` |
+| `AMI_ID` | TestDriver AMI ID | `ami-0504bf50fad62f312` |
+| `TD_API_KEY` | Your TestDriver API key | From [console.testdriver.ai](https://console.testdriver.ai) |
+
+<Tip>
+  Add these as **GitHub Repository Secrets** under Settings → Secrets and variables → Actions
+</Tip>
+
+### Example Workflows
+
+For complete production examples, see:
+- **[Scheduled Tests](https://github.com/testdriverai/testdriverai/blob/main/.github/workflows/acceptance-windows-scheduled.yaml)** — Daily automated test runs
+- **[Reusable Workflow](https://github.com/testdriverai/testdriverai/blob/main/.github/workflows/windows-self-hosted.yaml)** — Shared configuration for PR and scheduled tests
+
+## Advanced Usage
+
+### Using an Existing Instance
+
+If you already have a running instance, you can skip automatic spawning by providing `TD_IP`:
+
+```bash
+TD_OS=windows TD_IP=1.2.3.4 vitest run
+```
+
+The `setup-aws` hook will detect `TD_IP` is already set and skip spawning a new instance.
+
+### Manual Instance Management
+
+For advanced use cases, you can manually spawn instances using the [`spawn-runner.sh`](https://github.com/testdriverai/testdriverai/blob/main/setup/aws/spawn-runner.sh) script:
+
+```bash
+AWS_REGION=us-east-2 \
+AMI_ID=ami-0504bf50fad62f312 \
+AWS_LAUNCH_TEMPLATE_ID=lt-xxx \
+bash setup/aws/spawn-runner.sh
+```
+
+Output:
+```
+PUBLIC_IP=1.2.3.4
+INSTANCE_ID=i-1234567890abcdef0
+AWS_REGION=us-east-2
+```
+
+Then manually terminate when done:
+
+```bash
+aws ec2 terminate-instances \
+  --instance-ids i-1234567890abcdef0 \
+  --region us-east-2
+```
+
+For complete production examples, see:
+- **[Scheduled Tests](https://github.com/testdriverai/testdriverai/blob/main/.github/workflows/acceptance-windows-scheduled.yaml)** — Daily automated test runs
+- **[Reusable Workflow](https://github.com/testdriverai/testdriverai/blob/main/.github/workflows/windows-self-hosted.yaml)** — Shared configuration for PR and scheduled tests
+
+### Connecting to an Instance
+
+You can connect to running instances via:
+- **RDP** — Use the public IP on port 3389
+- **VNC** — Access via web browser at `http://<public-ip>:5900`
+- **AWS Console** — Use EC2 Instance Connect or Session Manager
+
+<Note>
+  Stopped instances retain their EBS volumes and can be restarted later. Terminated instances are permanently deleted. Always terminate instances when done to avoid storage costs.
+</Note>
+
+## AMI Customization
+
+The TestDriver Golden Image comes pre-configured with:
+
+- Windows Server with desktop environment
+- VNC + web server for remote access
+- Python, Node.js, Git
+- TestDriver agent and dependencies
+
+### Creating a Custom AMI
+
+You can customize the AMI to include additional software or configurations:
+
+<Steps>
+  <Step title="Connect via RDP">
+    Use the default credentials:
+    - **Username**: `testdriver`
+    - **Password**: `wwv9uJ0sqlulbN3`
+  </Step>
+  
+  <Step title="Change the Password">
+    **Critical**: Run the password rotation script immediately:
+    ```powershell
+    C:\testdriver\RotateLocalPasswords.ps1
+    ```
+    Save the new password securely.
+  </Step>
+  
+  <Step title="Install Your Software">
+    Install any additional dependencies, configure settings, or modify the environment as needed.
+  </Step>
+  
+  <Step title="Create New AMI">
+    Use the AWS console or CLI to create an AMI from your modified instance. Update your workflow to use the new AMI ID.
+  </Step>
+</Steps>
+
+<Warning>
+  **Security**: Never use the default password in production. Always rotate passwords before creating custom AMIs.
+</Warning>
+
+## Security Best Practices
+
+### Network Security
+
+- **Restrict CIDR blocks** — Only allow access from known IP ranges
+- **Use VPC endpoints** — For private AWS service communication
+- **Enable VPC Flow Logs** — For network monitoring
+
+### AWS Authentication
+
+Use OIDC instead of long-term credentials for GitHub Actions:
+
+```yaml
+permissions:
+  id-token: write
+  contents: read
+
+steps:
+  - name: Configure AWS credentials
+    uses: aws-actions/configure-aws-credentials@v4
+    with:
+      role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
+      aws-region: us-east-2
+```
+
+See [GitHub's OIDC documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect) for setup instructions.
+
+### Instance Security
+
+- **Terminate immediately** after tests complete
+- **Monitor costs** with AWS billing alerts
+- **Use least-privilege IAM roles**
+- **Enable CloudTrail** for audit logging
+
+## Troubleshooting
+
+### Instance Not Responding
+
+- **Check security groups** — Verify required ports are open (RDP 3389, VNC 5900, TestDriver ports)
+- **Verify status checks** — Ensure instance has passed AWS status checks
+- **Check AMI compatibility** — Some instance types don't support certain AMIs
+
+### Connection Timeouts
+
+- Verify network connectivity from CI runner to instance
+- Check VPC routing and internet gateway configuration
+- Confirm instance is in the correct subnet
+
+### AWS CLI Errors
+
+- Validate credentials and permissions
+- Check service quotas and limits
+- Verify region consistency across all resources

package/ai/skills/testdriver-cache/SKILL.md

@@ -0,0 +1,221 @@
+---
+name: testdriver:cache
+description: Speed up tests with screenshot-based caching
+---
+<!-- Generated from cache.mdx. DO NOT EDIT. -->
+
+## Overview
+
+The cache system speeds up repeated test runs by comparing screenshots to cached results. When the screen hasn't changed significantly, cached element positions are reused instead of making an AI call.
+
+Cache works at two levels:
+- **Screen cache** — pixel diff comparison between the current screenshot and the cached screenshot
+- **Element cache** — OpenCV template matching to verify the cached element position is still correct
+
+## How It Works
+
+1. On `find()`, the SDK sends the current screenshot and cache metadata to the API
+2. The API compares the screenshot against previously cached results for the same `cacheKey`
+3. If the screen pixel diff is within the `screen` threshold AND the element template match exceeds the `element` threshold, the cached position is returned
+4. Otherwise, a new AI call is made and the result is cached
+
+```mermaid
+flowchart LR
+    A[Screenshot + cacheKey] --> B{Screen Diff\n< threshold?}
+    B -- Yes --> C{Template\nMatch OK?}
+    C -- Yes --> D[Return cached position]
+    B -- No --> E[AI Call - fresh]
+    C -- No --> F[AI Call - fresh]
+```
+
+## Configuration
+
+### Constructor Options
+
+```javascript
+const testdriver = new TestDriver({
+  cache: {
+    enabled: true,
+    thresholds: {
+      find: {
+        screen: 0.05,     // 5% pixel diff allowed (default)
+        element: 0.8,     // 80% OpenCV correlation required (default)
+      },
+      assert: 0.05,       // 5% pixel diff for assertions (default)
+    },
+  },
+  cacheKey: 'my-custom-key',   // overrides auto-generated key
+});
+```
+
+<ParamField path="cache" type="CacheConfig | false">
+  Cache configuration object, or `false` to disable entirely.
+
+  <Expandable title="properties">
+    <ParamField path="enabled" type="boolean" default={true}>
+      Enable or disable the cache system. Requires a valid `cacheKey` to actually activate.
+    </ParamField>
+    
+    <ParamField path="thresholds" type="CacheThresholds">
+      Threshold configuration for different command types.
+
+      <Expandable title="properties">
+        <ParamField path="find" type="FindCacheThresholds">
+          Thresholds for `find()` and `findAll()`.
+
+          <Expandable title="properties">
+            <ParamField path="screen" type="number" default={0.05}>
+              Maximum pixel diff percentage allowed between the current screenshot and the cached screenshot. Lower values require a closer match. Range: `0` to `1`.
+            </ParamField>
+            
+            <ParamField path="element" type="number" default={0.8}>
+              Minimum OpenCV template matching correlation required for the cached element crop. Higher values require a closer match. Range: `0` to `1`. Only used for `find()`, not `findAll()`.
+            </ParamField>
+          </Expandable>
+        </ParamField>
+        
+        <ParamField path="assert" type="number" default={0.05}>
+          Maximum pixel diff allowed for assertion cache hits.
+        </ParamField>
+      </Expandable>
+    </ParamField>
+  </Expandable>
+</ParamField>
+
+<ParamField path="cacheKey" type="string">
+  Unique key for cache lookups. If not provided, an auto-generated key is created from a SHA-256 hash of the calling test file (first 16 hex characters). The cache key changes automatically when your test file changes, providing automatic cache invalidation.
+</ParamField>
+
+### Disabling Cache
+
+```javascript
+// Via constructor
+const testdriver = new TestDriver({ cache: false });
+
+// Via environment variable
+// TD_NO_CACHE=true npx vitest run
+```
+
+When cache is disabled, all thresholds are set to `-1` internally, causing the API to skip cache lookups.
+
+### Per-Command Overrides
+
+Override cache thresholds for individual commands:
+
+```javascript
+// Stricter screen matching for this specific find
+const el = await testdriver.find('submit button', {
+  cache: {
+    thresholds: { screen: 0.01, element: 0.95 },
+  },
+});
+
+// Custom cache key for a specific assertion
+await testdriver.assert('dashboard loaded', {
+  cache: { threshold: 0.01 },
+  cacheKey: 'dashboard-check',
+});
+```
+
+## Threshold Priority
+
+Thresholds are resolved in priority order (highest wins):
+
+| Priority | Source | Example |
+|----------|--------|---------|
+| 1 (highest) | Per-command option | `find(desc, { cache: { thresholds: { screen: 0.1 } } })` |
+| 2 | Legacy number argument | `find(desc, 0.1)` |
+| 3 | Global constructor config | `new TestDriver({ cache: { thresholds: { find: { screen: 0.1 } } } })` |
+| 4 (lowest) | Hard-coded defaults | `screen: 0.05`, `element: 0.8`, `assert: 0.05` |
+
+## Auto-Generated Cache Key
+
+When you don't specify a `cacheKey`, the SDK automatically generates one:
+
+1. Walks the call stack to find your test file
+2. Reads the file content
+3. Computes a **SHA-256 hash** of the content
+4. Uses the **first 16 hex characters** as the cache key
+
+This means:
+- **Same test file** → same cache key → cache hits
+- **Modified test file** → different hash → automatic cache invalidation
+- **Different test files** → different keys → isolated caches
+
+```javascript
+// Auto-generated cache key from test file hash
+const testdriver = new TestDriver();
+// cacheKey = "a3f2b1c4d5e6f7a8" (auto)
+
+// Manual override
+const testdriver = new TestDriver({ cacheKey: 'login-test-v2' });
+```
+
+## Template Matching (OpenCV)
+
+Element cache validation uses OpenCV's normalized cross-correlation coefficient (`TM_CCOEFF_NORMED`) to verify that the cached element is still visible at the expected position.
+
+**Algorithm:**
+1. Load the cached element crop (needle) and current screenshot (haystack)
+2. Run `cv.matchTemplate()` with `TM_CCOEFF_NORMED`
+3. Binary threshold at the configured element threshold
+4. Find contours to extract match positions
+5. Return matches with `{ x, y, width, height, centerX, centerY }`
+
+**Scale factors tried:** `[1, 0.5, 2, 0.75, 1.25, 1.5]`
+**Thresholds tried:** `[0.9, 0.8, 0.7]` (picks highest matching threshold)
+
+This accounts for minor scaling differences between screenshots taken at different times or resolutions.
+
+## Cache Partitioning
+
+Cache entries are partitioned by:
+- **`cacheKey`** — identifies the test file
+- **`os`** — operating system (linux, windows, darwin)
+- **`resolution`** — screen resolution
+
+This means cache from a Linux run won't be used for a Windows run, even with the same cache key.
+
+## Debugging Cache
+
+API responses include cache metadata:
+
+| Field | Description |
+|---|---|
+| `cacheHit` | `true` if cache was used |
+| `similarity` | Pixel diff percentage between screenshots |
+| `cacheSimilarity` | OpenCV template match score |
+
+Use `getDebugInfo()` on an element to inspect cache results:
+
+```javascript
+const el = await testdriver.find('submit button');
+const debug = el.getDebugInfo();
+console.log(debug);
+// { cacheHit: true, similarity: 0.02, cacheSimilarity: 0.92, ... }
+```
+
+## Types
+
+```typescript
+interface CacheConfig {
+  enabled?: boolean;            // Default: true
+  thresholds?: CacheThresholds;
+}
+
+interface CacheThresholds {
+  find?: FindCacheThresholds;
+  assert?: number;              // Default: 0.05
+}
+
+interface FindCacheThresholds {
+  screen?: number;              // Default: 0.05
+  element?: number;             // Default: 0.8
+}
+
+interface CacheDebugInfo {
+  cacheHit: boolean;
+  similarity: number;
+  cacheSimilarity: number;
+}
+```