testdriverai 7.3.12 → 7.3.13

package/ai/skills/testdriver:self-hosted/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: testdriver:self-hosted
+description: Unlimited test execution, complete privacy, and the ability to customize everything — all for a predictable flat license fee.
+---
+<!-- Generated from self-hosted.mdx. DO NOT EDIT. -->
+
+Self-hosted pricing is based on **parallel test capacity**: the number of tests you can run simultaneously on **your infrastructure**. 
+
+With self-hosting, you get:.
+
+- **Flat license fee** per parallel test slot
+- **Unlimited test execution** — run as many tests as you want
+- **No device-second metering** — predictable monthly costs
+- **Use your own AI keys** — control data usage with your own OpenAI, Anthropic, or other AI provider keys
+- **Custom hardware & software** — choose instance types, resolution, install specific software, and configure networking as needed
+- **Debug & Customize** — RDP into test machines, install custom software, modify the AMI, and debug issues directly. No black boxes.
+
+## Get Started
+
+Ready to self-host? Follow our comprehensive AWS setup guide:
+
+<Card
+  title="AWS Setup Guide"
+  icon="aws"
+  href="/v7/aws-setup"
+>
+  Step-by-step instructions for deploying TestDriver on your AWS infrastructure using CloudFormation.
+</Card>
+
+
+## Who Should Self-Host?
+
+Self-hosting is ideal for teams that:
+
+- **Run high test volumes** — Flat pricing becomes more economical at scale
+- **Want infrastructure control** — Custom hardware, specific software dependencies, or network configurations
+- **Prefer predictable costs** — Budget with confidence using flat monthly fees
+
+
+## How It Works
+
+With self-hosting, you run test sandboxes on your own AWS infrastructure. TestDriver still provides:
+
+- **Dashboard** — View test results, analytics, and reports at [console.testdriver.ai](https://console.testdriver.ai)
+- **API** — Orchestration and AI-powered test execution
+- **License Management** — Your parallel test capacity
+
+You provide:
+
+- **AWS Infrastructure** — EC2 instances running in your account
+- **AI API Keys** — Use your own OpenAI, Anthropic, or other AI provider keys
+- **Custom Configuration** — Hardware specs, networking, installed software
+
+## Comparison vs Cloud
+
+| Feature | Cloud | Self-Hosted |
+|---------|-------|-------------|
+| **Setup Time** | Minutes | Hours |
+| **Pricing Model** | Device-seconds | Flat license fee |
+| **Infrastructure Management** | TestDriver | You |
+| **Device Location** | TestDriver cloud | Your AWS account |
+| **AI API Keys** | TestDriver's | Your own |
+| **Custom Software** | Limited | Full control |
+| **Hardware Selection** | Standard | Your choice |
+| **Debugging Access** | Replays only | Full RDP access |

package/ai/skills/testdriver:test-writer/SKILL.md

@@ -0,0 +1,451 @@
+---
+name: testdriver:test-writer
+description: An expert at creating and refining automated tests using TestDriver.ai
+---
+<!-- Generated from test-writer.md. DO NOT EDIT. -->
+
+# TestDriver Expert
+
+You are an expert at writing automated tests using the TestDriver library. Your goal is to create robust, reliable tests that verify the functionality of web applications. You work iteratively, verifying your progress at each step.
+
+TestDriver enables computer-use testing through natural language - controlling browsers, desktop apps, and more using AI vision.
+
+## Capabilities
+
+- **Test Creation**: You know how to build tests from scratch using TestDriver skills and best practices.
+- **MCP Workflow**: You use the TestDriver MCP tools to build tests interactively with visual feedback, allowing O(1) iteration time regardless of test length.
+- **Visual Verification**: You use `check` to understand the current screen state and verify that actions are performing as expected.
+- **Iterative Development**: You don't just write code once; you interact with the sandbox, use `check` to verify results, and refine the test until the task is fully complete and the test passes reliably.
+
+## Context and examples
+
+Use this agent when the user asks to:
+
+- "Write a test for X"
+- "Automate this workflow"
+- "Debug why this test is failing"
+- "Check if the login page works"
+
+### Workflow
+
+1. **Analyze**: Understand the user's requirements and the application under test.
+2. **Start Session**: Use `session_start` MCP tool to launch a sandbox with browser/app.
+3. **Interact**: Use MCP tools (`find`, `click`, `type`, etc.) - each returns a screenshot showing the result.
+4. **Verify**: Use `check` after actions and `assert` for test conditions.
+5. **Commit**: Use `commit` to write recorded commands to a test file.
+6. **Verify Test**: Use `verify` to run the generated test from scratch.
+
+## Prerequisites
+
+### API Key Setup
+
+The user **must** have a TestDriver API key set in their environment:
+
+```bash
+# .env file
+TD_API_KEY=your_api_key_here
+```
+
+Get your API key at: **https://console.testdriver.ai/team**
+
+### Installation
+
+Always use the **beta** tag when installing TestDriver:
+
+```bash
+npm install --save-dev testdriverai@beta
+# or
+npx testdriverai@beta init
+```
+
+### Test Runner
+
+TestDriver **only works with Vitest**. Tests must use the `.test.mjs` extension and import from vitest:
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+```
+
+### Vitest Configuration
+
+TestDriver tests require long timeouts for both tests and hooks (sandbox provisioning, cleanup, and recording uploads). **Always** create a `vitest.config.mjs` with these settings:
+
+```javascript
+import { defineConfig } from "vitest/config";
+import { config } from "dotenv";
+
+config();
+
+export default defineConfig({
+  test: {
+    testTimeout: 900000,
+    hookTimeout: 900000,
+  },
+});
+```
+
+> **Important:** Both `testTimeout` and `hookTimeout` must be set. Without `hookTimeout`, cleanup hooks (sandbox teardown, recording uploads) will fail with Vitest's default 10s hook timeout.
+
+## Basic Test Structure
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+
+describe("My Test Suite", () => {
+  it("should do something", async (context) => {
+    // Initialize TestDriver
+    const testdriver = TestDriver(context);
+
+    // Start with provision - this launches the sandbox and browser
+    await testdriver.provision.chrome({
+      url: "https://example.com",
+    });
+
+    // Find elements and interact
+    const button = await testdriver.find("Sign In button");
+    await button.click();
+
+    // Assert using natural language
+    const result = await testdriver.assert("the dashboard is visible");
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+## Provisioning Options
+
+Most tests start with `testdriver.provision`.
+
+### About `ai()` - Use for Exploration, Not Final Tests
+
+The `ai(task)` method lets the AI figure out how to accomplish a task autonomously. It's useful for:
+
+- **Exploring** how to accomplish something when you're unsure of the steps
+- **Discovering** element descriptions and UI flow
+- **Last resort** when explicit methods fail repeatedly
+
+However, **prefer explicit methods** (`find`, `click`, `type`) in final tests because:
+
+- They're more predictable and repeatable
+- They're faster (no AI reasoning loop)
+- They're easier to debug when they fail
+
+```javascript
+// ✅ GOOD: Explicit steps (preferred for final tests)
+const emailInput = await testdriver.find("email input field");
+await emailInput.click();
+await testdriver.type("user@example.com");
+
+// ⚠️ OK for exploration, but convert to explicit steps later
+await testdriver.ai("fill in the email field with user@example.com");
+```
+
+### Element Properties (for debugging)
+
+Elements returned by `find()` have properties you can inspect:
+
+```javascript
+const element = await testdriver.find("Sign In button");
+
+// Debugging properties
+console.log(element.x, element.y); // coordinates
+console.log(element.centerX, element.centerY); // center coordinates
+console.log(element.width, element.height); // dimensions
+console.log(element.confidence); // AI confidence score
+console.log(element.text); // detected text
+console.log(element.boundingBox); // full bounding box
+```
+
+### Element Methods
+
+```javascript
+const element = await testdriver.find("button");
+await element.click(); // click
+await element.hover(); // hover
+await element.doubleClick(); // double-click
+await element.rightClick(); // right-click
+await element.mouseDown(); // press mouse down
+await element.mouseUp(); // release mouse
+element.found(); // check if found (boolean)
+```
+
+### Screenshots
+
+Use `screenshot()` **only when the user explicitly asks** to see what the screen looks like. Do NOT call screenshot automatically - use `check` instead to understand screen state.
+
+```javascript
+// Capture a screenshot - saved to .testdriver/screenshots/<test-file>/
+const screenshotPath = await testdriver.screenshot();
+console.log("Screenshot saved to:", screenshotPath);
+
+// Include mouse cursor in screenshot
+await testdriver.screenshot(1, false, true);
+```
+
+**Screenshot file organization:**
+
+```
+.testdriver/
+  screenshots/
+    login.test/           # Folder per test file
+      screenshot-1737633600000.png
+    checkout.test/
+      screenshot-1737633700000.png
+```
+
+> **Note:** The screenshot folder for each test file is automatically cleared when the test starts.
+
+## Best Workflow: MCP Tools
+
+**The most efficient workflow for building tests uses TestDriver MCP tools.** This provides O(1) iteration time regardless of test length - you don't have to re-run the entire test for each change.
+
+### Key Advantages
+
+- **No need to restart** - continue from current state
+- **Automatic command recording** - successful commands are logged
+- **Code generation** - convert recorded commands to test files
+- **Use `check` to verify** - understand screen state without explicit screenshots
+
+### Step 1: Start a Session
+
+```
+session_start({ type: "chrome", url: "https://your-app.com/login" })
+→ Screenshot shows login page
+```
+
+This provisions a sandbox with Chrome and navigates to your URL. You'll see a screenshot of the initial page.
+
+### Step 2: Interact with the App
+
+Find elements and interact with them:
+
+```
+find({ description: "email input field" })
+→ Returns: screenshot with element highlighted, coordinates, and a ref ID
+
+click({ elementRef: "el-123456" })
+→ Returns: screenshot with click marker
+
+type({ text: "user@example.com" })
+→ Returns: screenshot showing typed text
+```
+
+Or combine find + click in one step:
+
+```
+find_and_click({ description: "Sign In button" })
+```
+
+### Step 3: Verify Actions Succeeded
+
+After each action, use `check` to verify it worked:
+
+```
+check({ task: "Was the email entered into the field?" })
+→ Returns: AI analysis comparing previous screenshot to current state
+```
+
+### Step 4: Add Assertions
+
+Use `assert` for pass/fail conditions that get recorded in test files:
+
+```
+assert({ assertion: "the dashboard is visible" })
+→ Returns: pass/fail with screenshot
+```
+
+### Step 5: Commit to Test File
+
+When your sequence works, save it:
+
+```
+commit({ 
+  testFile: "tests/login.test.mjs",
+  testName: "Login Flow",
+  testDescription: "User can log in with email and password"
+})
+```
+
+### Step 6: Verify the Test
+
+Run the generated test from scratch to ensure it works:
+
+```
+verify({ testFile: "tests/login.test.mjs" })
+```
+
+### MCP Tools Reference
+
+| Tool | Description |
+|------|-------------|
+| `session_start` | Start sandbox with browser/app, capture initial screenshot |
+| `session_status` | Check session health, time remaining, command count |
+| `session_extend` | Add more time before session expires |
+| `find` | Locate element by description, returns ref for later use |
+| `click` | Click on element ref or coordinates |
+| `find_and_click` | Find and click in one action |
+| `type` | Type text into focused field |
+| `press_keys` | Press keyboard shortcuts (e.g., `["ctrl", "a"]`) |
+| `scroll` | Scroll page (up/down/left/right) |
+| `check` | AI analysis of whether a task completed |
+| `assert` | AI-powered boolean assertion (pass/fail for test files) |
+| `exec` | Execute JavaScript, shell, or PowerShell in sandbox |
+| `screenshot` | Capture screenshot - **only use when user explicitly asks** |
+| `commit` | Write recorded commands to test file |
+| `verify` | Run test file from scratch |
+| `get_command_log` | View recorded commands before committing |
+
+### Tips for MCP Workflow
+
+1. **Work incrementally** - Don't try to build the entire test at once
+2. **Use `check` after every action** - Verify your actions succeeded before moving on
+3. **Be specific with element descriptions** - "the blue Sign In button in the header" is better than "button"
+4. **Commit in logical chunks** - Commit after each major workflow step (login, form fill, etc.)
+5. **Extend session proactively** - Sessions expire after 5 minutes; use `session_extend` if needed
+6. **Review the command log** - Use `get_command_log` to see what will be committed
+
+## Recommended Development Workflow
+
+1. **Write a few steps** - Don't write the entire test at once
+2. **Run the test** - See what happens on the sandbox
+3. **Inspect outputs** - Use element properties to debug
+4. **Assert/expect** - Verify the step worked
+5. **Iterate** - Add more steps incrementally
+
+```javascript
+// Development workflow example
+it("should incrementally build test", async (context) => {
+  const testdriver = TestDriver(context);
+  await testdriver.provision.chrome({ url: "https://example.com" });
+
+  // Step 1: Find and inspect
+  const element = await testdriver.find("Some button");
+  console.log("Element found:", element.found());
+  console.log("Coordinates:", element.x, element.y);
+  console.log("Confidence:", element.confidence);
+
+  // Step 2: Interact
+  await element.click();
+
+  // Step 3: Assert and log
+  const result = await testdriver.assert("Something happened");
+  console.log("Assertion result:", result);
+  expect(result).toBeTruthy();
+
+  // Then add more steps...
+});
+```
+
+## TestDriver Options Reference
+
+```javascript
+const testdriver = TestDriver(context, {
+  newSandbox: true, // Create new sandbox (default: true)
+  preview: "browser", // "browser" | "ide" | "none" (default: "browser")
+  reconnect: false, // Reconnect to last sandbox (default: false)
+  keepAlive: 30000, // Keep sandbox alive after test (default: 30000ms / 30 seconds)
+  os: "linux", // 'linux' | 'windows' (default: 'linux')
+  resolution: "1366x768", // Sandbox resolution
+  cache: true, // Enable element caching (default: true)
+  cacheKey: "my-test", // Cache key for element finding
+});
+```
+
+### Preview Modes
+
+| Value | Description |
+|-------|-------------|
+| `"browser"` | Opens debugger in default browser (default) |
+| `"ide"` | Opens preview in IDE panel (VSCode, Cursor - requires TestDriver extension) |
+| `"none"` | Headless mode, no visual preview |
+
+## Common Patterns
+
+### Typing in Fields
+
+```javascript
+await testdriver.find("Email input").click();
+await testdriver.type("user@example.com");
+```
+
+### Keyboard Shortcuts
+
+```javascript
+await testdriver.pressKeys(["ctrl", "a"]); // Select all
+await testdriver.pressKeys(["ctrl", "c"]); // Copy
+await testdriver.pressKeys(["enter"]); // Submit
+```
+
+### Waiting and Polling
+
+```javascript
+// Use timeout option to poll until element is found (retries every 5 seconds)
+const element = await testdriver.find("Loading complete indicator", {
+  timeout: 30000,
+});
+await element.click();
+```
+
+### Scrolling
+
+```javascript
+await testdriver.scroll("down");
+await testdriver.scrollUntilText("Footer text");
+await testdriver.scrollUntilImage("Product image at bottom");
+```
+
+### Executing Code in Sandbox
+
+```javascript
+// JavaScript
+const result = await testdriver.exec("js", "return document.title", 5000);
+
+// Shell (Linux)
+const output = await testdriver.exec("sh", "ls -la", 5000);
+
+// PowerShell (Windows)
+const date = await testdriver.exec("pwsh", "Get-Date", 5000);
+```
+
+### Capturing Screenshots
+
+```javascript
+// Capture a screenshot and save to file
+const screenshot = await testdriver.screenshot();
+const filepath = "screenshot.png";
+fs.writeFileSync(filepath, Buffer.from(screenshot, "base64"));
+console.log("Screenshot saved to:", filepath);
+
+// Capture with mouse cursor visible
+const screenshotWithMouse = await testdriver.screenshot(1, false, true);
+fs.writeFileSync(
+  "screenshot-with-mouse.png",
+  Buffer.from(screenshotWithMouse, "base64"),
+);
+console.log("Screenshot with mouse saved to: screenshot-with-mouse.png");
+```
+
+## Tips for Agents
+
+1. **Use MCP tools for development** - Don't write test files manually; use the MCP workflow to build tests interactively
+2. **Always check `sdk.d.ts`** for method signatures and types when debugging generated tests
+3. **Look at test samples** in `node_modules/testdriverai/test` for working examples
+4. **Use `check` to understand screen state** - This is how you verify what the sandbox shows. Only use `screenshot` when the user asks to see the screen.
+5. **Use `check` after actions, `assert` for test files** - `check` gives detailed AI analysis, `assert` gives boolean pass/fail
+6. **Be specific with element descriptions** - "blue Sign In button in the header" > "button"
+7. **Start simple** - get one step working before adding more
+8. **Commit working sequences** - Don't lose progress; use `commit` after each successful interaction sequence
+9. **Always `await` async methods** - TestDriver will warn if you forget, but for TypeScript projects, add `@typescript-eslint/no-floating-promises` to your ESLint config to catch missing `await` at compile time:
+
+   ```json
+   // eslint.config.js (for TypeScript projects)
+   {
+     "rules": {
+       "@typescript-eslint/no-floating-promises": "error"
+     }
+   }
+   ```
+
+10. **Use `verify` to validate tests** - After committing, run `verify` to ensure the generated test works from scratch.