testdriverai 7.2.3 → 7.2.10

package/agents.md

@@ -0,0 +1,393 @@
+# TestDriver Agent Guide
+
+This guide is designed for AI agents working with TestDriver. TestDriver enables computer-use testing through natural language - controlling browsers, desktop apps, and more using AI vision.
+
+## Quick Reference
+
+| Resource | Location |
+|----------|----------|
+| Code samples | `node_modules/testdriverai/test` |
+| TypeScript types | `node_modules/testdriverai/sdk.d.ts` |
+| Documentation | `node_modules/testdriverai/docs` |
+| API Key | [console.testdriver.ai/team](https://console.testdriver.ai/team) |
+
+## Prerequisites
+
+### API Key Setup
+
+The user **must** have a TestDriver API key set in their environment:
+
+```bash
+# .env file
+TD_API_KEY=your_api_key_here
+```
+
+Get your API key at: **https://console.testdriver.ai/team**
+
+### Test Runner
+
+TestDriver **only works with Vitest**. Tests must use the `.test.mjs` extension and import from vitest:
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+```
+
+## Basic Test Structure
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("My Test Suite", () => {
+  it("should do something", async (context) => {
+    // Initialize TestDriver with options
+    const testdriver = TestDriver(context, { newSandbox: true, headless: false });
+    
+    // Start with provision - this launches the sandbox and browser
+    await testdriver.provision.chrome({
+      url: 'https://example.com',
+    });
+
+    // Find elements and interact
+    const button = await testdriver.find("Sign In button");
+    await button.click();
+
+    // Assert using natural language
+    const result = await testdriver.assert("the dashboard is visible");
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+## Provisioning Options
+
+Most tests start with `testdriver.provision`. Choose the right one:
+
+### `provision.chrome` - Web Testing (Most Common)
+```javascript
+await testdriver.provision.chrome({
+  url: 'https://your-app.com',
+});
+```
+
+### `provision.installer` - Desktop App Testing
+```javascript
+// Download and install an application
+const filePath = await testdriver.provision.installer({
+  url: 'https://example.com/app.deb',  // or .msi, .exe, .sh
+  launch: true,  // auto-launch after install
+});
+```
+
+## Key SDK Methods
+
+The SDK has TypeScript types in `sdk.d.ts`. Key methods:
+
+| Method | Purpose |
+|--------|---------|
+| `find(description)` | Find element by natural language |
+| `findAll(description)` | Find all matching elements |
+| `assert(assertion)` | AI-powered assertion |
+| `type(text)` | Type text |
+| `pressKeys([keys])` | Press keyboard keys |
+| `scroll(direction)` | Scroll the page |
+| `exec(language, code)` | Execute code in sandbox |
+| `ai(task)` | AI exploratory loop (see note below) |
+
+### About `ai()` - Use for Exploration, Not Final Tests
+
+The `ai(task)` method lets the AI figure out how to accomplish a task autonomously. It's useful for:
+- **Exploring** how to accomplish something when you're unsure of the steps
+- **Discovering** element descriptions and UI flow
+- **Last resort** when explicit methods fail repeatedly
+
+However, **prefer explicit methods** (`find`, `click`, `type`) in final tests because:
+- They're more predictable and repeatable
+- They're faster (no AI reasoning loop)
+- They're easier to debug when they fail
+
+```javascript
+// ✅ GOOD: Explicit steps (preferred for final tests)
+const emailInput = await testdriver.find("email input field");
+await emailInput.click();
+await testdriver.type("user@example.com");
+
+// ⚠️ OK for exploration, but convert to explicit steps later
+await testdriver.ai("fill in the email field with user@example.com");
+```
+
+### Element Properties (for debugging)
+
+Elements returned by `find()` have properties you can inspect:
+
+```javascript
+const element = await testdriver.find("Sign In button");
+
+// Debugging properties
+console.log(element.x, element.y);           // coordinates
+console.log(element.centerX, element.centerY); // center coordinates  
+console.log(element.width, element.height);  // dimensions
+console.log(element.confidence);             // AI confidence score
+console.log(element.text);                   // detected text
+console.log(element.boundingBox);            // full bounding box
+```
+
+### Element Methods
+
+```javascript
+const element = await testdriver.find("button");
+await element.click();        // click
+await element.hover();        // hover
+await element.doubleClick();  // double-click
+await element.rightClick();   // right-click
+await element.mouseDown();    // press mouse down
+await element.mouseUp();      // release mouse
+element.found();              // check if found (boolean)
+```
+
+## Best Workflow: Two-File Pattern
+
+**The most efficient workflow for building tests uses two files.** This prevents having to restart from scratch when experimenting with new steps.
+
+### IMPORTANT: When to Use This Pattern
+
+- **If a working test already exists**: Only create an `experiment.test.mjs` file to add new steps. Do NOT recreate the stable file.
+- **If starting from scratch**: Start with a MINIMAL setup file, run it, verify it passes, THEN create the experiment file.
+
+### Step 1: Create a Minimal Setup File
+
+Start with the bare minimum - just provision and one assertion. **Do NOT call it "stable" yet** - name it `setup.test.mjs` until it's proven to work:
+
+```javascript
+/**
+ * Setup file - MINIMAL steps to get to starting state
+ * Only add more steps AFTER this passes!
+ */
+import { afterAll, describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("Setup State", () => {
+
+  afterAll(async () => {
+    // DO NOT disconnect - keep sandbox alive for reconnect
+    console.log("Sandbox staying alive for 30 seconds (keepAlive)");
+  });
+
+  it("should set up the application state", async (context) => {
+    const testdriver = TestDriver(context, { 
+      newSandbox: true, 
+      headless: false 
+    });
+    
+    await testdriver.provision.chrome({
+      url: 'https://your-app.com/login',
+    });
+
+    // Start with just ONE assertion to verify we're on the right page
+    const result = await testdriver.assert("I can see the login page");
+    expect(result).toBeTruthy();
+    
+    console.log("✅ Setup ready - run experiment.test.mjs now");
+  });
+});
+```
+
+### Step 2: Run Setup File and Verify It Passes
+
+```bash
+vitest run tests/setup.test.mjs
+```
+
+**Only proceed to Step 3 if this passes!** If it fails, fix it first.
+
+### Step 3: Create Experiment File
+
+Only AFTER the setup file passes, create the experiment file.
+
+**CRITICAL: The experiment file must NOT call `provision`!** It reconnects to the existing sandbox where provision already ran:
+
+```javascript
+/**
+ * Experiment file - reconnects to existing sandbox
+ * Run AFTER setup.test.mjs passes (within 2 minutes)
+ */
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("Experiment", () => {
+
+  it("should continue from existing state", async (context) => {
+    const testdriver = TestDriver(context, { 
+      newSandbox: true, 
+      headless: false,
+      reconnect: true  // ← Key: reconnects to last sandbox
+    });
+    
+    // NO provision here! The sandbox is already running from setup.test.mjs
+    
+    // Experiment with new steps here - try ONE thing at a time
+    const element = await testdriver.find("email input");
+    console.log("Found element:", element.found(), element.getCoordinates());
+    
+    await element.click();
+    await testdriver.type("user@example.com");
+    
+    // Assert after each major step
+    const result = await testdriver.assert("email is filled in");
+    console.log("Assertion result:", result);
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+### Step 4: Iterate in Experiment File
+
+- Run experiment, see output, fix issues
+- Once steps work, move them to the setup file
+- Re-run setup file to verify it still passes
+- Repeat until complete
+
+### Running the Two-File Pattern
+
+```bash
+# Step 1: Run setup file (must pass first!)
+vitest run tests/setup.test.mjs
+
+# Step 2: Within 2 minutes, run experiment file
+vitest run tests/experiment.test.mjs
+```
+
+### After Experimentation: Combine and Rename
+
+**IMPORTANT:** Once the test is working, combine both files into a single, properly-named test file:
+
+1. **Merge the code** - Copy the working steps from `experiment.test.mjs` into `setup.test.mjs`
+2. **Remove reconnect options** - Delete `reconnect: true` since the final test runs standalone
+3. **Rename the file** - Give it a meaningful name like `login-flow.test.mjs` or `checkout-process.test.mjs`
+4. **Delete the experiment file** - Clean up `experiment.test.mjs`
+
+```javascript
+// Final combined test: login-flow.test.mjs
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("Login Flow", () => {
+  it("should log in and open settings", async (context) => {
+    const testdriver = TestDriver(context, { newSandbox: true, headless: false });
+    
+    await testdriver.provision.chrome({ url: 'https://your-app.com/login' });
+
+    // Steps from stable file
+    await testdriver.find("email input").click();
+    await testdriver.type("user@example.com");
+    await testdriver.find("password input").click();
+    await testdriver.type("password123");
+    await testdriver.find("Login button").click();
+    
+    // Steps from experiment file
+    const settingsBtn = await testdriver.find("Settings button");
+    await settingsBtn.click();
+    
+    const result = await testdriver.assert("Settings panel is open");
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+## Recommended Development Workflow
+
+1. **Write a few steps** - Don't write the entire test at once
+2. **Run the test** - See what happens on the sandbox
+3. **Inspect outputs** - Use element properties to debug
+4. **Assert/expect** - Verify the step worked
+5. **Iterate** - Add more steps incrementally
+
+```javascript
+// Development workflow example
+it("should incrementally build test", async (context) => {
+  const testdriver = TestDriver(context, { newSandbox: true });
+  await testdriver.provision.chrome({ url: 'https://example.com' });
+
+  // Step 1: Find and inspect
+  const element = await testdriver.find("Some button");
+  console.log("Element found:", element.found());
+  console.log("Coordinates:", element.x, element.y);
+  console.log("Confidence:", element.confidence);
+  
+  // Step 2: Interact
+  await element.click();
+  
+  // Step 3: Assert and log
+  const result = await testdriver.assert("Something happened");
+  console.log("Assertion result:", result);
+  expect(result).toBeTruthy();
+  
+  // Then add more steps...
+});
+```
+
+## TestDriver Options Reference
+
+```javascript
+const testdriver = TestDriver(context, {
+  newSandbox: true,       // Create new sandbox (default: true)
+  headless: false,        // Run in headless mode (default: false)
+  reconnect: false,       // Reconnect to last sandbox (default: false)
+  keepAlive: 30000,       // Keep sandbox alive after test (default: 30000ms / 30 seconds)
+  os: 'linux',            // 'linux' | 'windows' (default: 'linux')
+  resolution: '1366x768', // Sandbox resolution
+  cache: true,            // Enable element caching (default: true)
+  cacheKey: 'my-test',    // Cache key for element finding
+});
+```
+
+## Common Patterns
+
+### Typing in Fields
+```javascript
+await testdriver.find("Email input").click();
+await testdriver.type("user@example.com");
+```
+
+### Keyboard Shortcuts
+```javascript
+await testdriver.pressKeys(["ctrl", "a"]);  // Select all
+await testdriver.pressKeys(["ctrl", "c"]);  // Copy
+await testdriver.pressKeys(["enter"]);       // Submit
+```
+
+### Waiting and Polling
+```javascript
+// Use timeout option to poll until element is found (retries every 5 seconds)
+const element = await testdriver.find("Loading complete indicator", { timeout: 30000 });
+await element.click();
+```
+
+### Scrolling
+```javascript
+await testdriver.scroll("down");
+await testdriver.scrollUntilText("Footer text");
+await testdriver.scrollUntilImage("Product image at bottom");
+```
+
+### Executing Code in Sandbox
+```javascript
+// JavaScript
+const result = await testdriver.exec("js", "return document.title", 5000);
+
+// Shell (Linux)
+const output = await testdriver.exec("sh", "ls -la", 5000);
+
+// PowerShell (Windows)
+const date = await testdriver.exec("pwsh", "Get-Date", 5000);
+```
+
+## Tips for Agents
+
+1. **Always check `sdk.d.ts`** for method signatures and types
+2. **Look at test samples** in `node_modules/testdriverai/test` for working examples
+3. **Use reconnect pattern** when iterating on test steps
+4. **Log element properties** to understand what the AI sees
+5. **Use `assert()` with specific, descriptive natural language**
+6. **Start simple** - get one step working before adding more

package/bin/testdriverai.js

@@ -1,5 +1,8 @@
 #!/usr/bin/env node
 
+// Initialize Sentry first, before any other modules
+const sentry = require("../lib/sentry");
+
 // Set process priority if possible
 const os = require("os");
 try {
@@ -10,5 +13,10 @@ try {
   // Ignore if not permitted
 }
 
+// Ensure Sentry flushes on exit
+process.on("beforeExit", async () => {
+  await sentry.flush();
+});
+
 // Run the CLI
 require("../interfaces/cli.js");

package/debugger/index.html

@@ -307,9 +307,46 @@
         text-align: center;
         user-select: none;
       }
+
+      .close-button {
+        position: fixed;
+        top: 12px;
+        right: 12px;
+        z-index: 100;
+        background: rgba(0, 0, 0, 0.8);
+        border: 1px solid #444;
+        color: #fff;
+        padding: 8px 16px;
+        border-radius: 6px;
+        cursor: pointer;
+        font-size: 13px;
+        font-weight: 500;
+        pointer-events: auto;
+        transition: all 0.2s ease;
+        display: flex;
+        align-items: center;
+        gap: 6px;
+      }
+
+      .close-button:hover {
+        background: rgba(220, 53, 69, 0.9);
+        border-color: #dc3545;
+      }
+
+      .close-button svg {
+        width: 14px;
+        height: 14px;
+        fill: currentColor;
+      }
     </style>
   </head>
   <body>
+    <!-- Close window button -->
+    <button class="close-button" onclick="window.close()" title="Close this window">
+      <svg viewBox="0 0 24 24"><path d="M19 6.41L17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12z"/></svg>
+      Close
+    </button>
+
     <!-- Loading screen -->
     <div class="loading-screen" id="loading-screen">
       <div class="testdriver-logo">