@testdriverai/agent 7.9.32-test → 7.9.34-canary

package/ai/skills/testdriver:parse/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: testdriver:parse
+description: Detect all UI elements on screen using OmniParser
+---
+<!-- Generated from parse.mdx. DO NOT EDIT. -->
+
+## Overview
+
+Parse the current screen using OmniParser v2 to detect all visible UI elements. Returns structured data including element types, text content, interactivity levels, and bounding box coordinates.
+
+This method analyzes the entire screen and returns every detected element. It's useful for:
+- Understanding the full UI layout of a screen
+- Finding all clickable or interactive elements
+- Building custom element-based logic
+- Debugging what elements TestDriver can detect
+- Accessibility auditing
+
+<Note>
+  **Availability**: `parse()` requires an enterprise or self-hosted plan. It uses OmniParser v2 server-side for element detection.
+</Note>
+
+## Syntax
+
+```javascript
+const result = await testdriver.parse()
+```
+
+## Parameters
+
+None.
+
+## Returns
+
+`Promise<ParseResult>` - Object containing detected UI elements
+
+### ParseResult
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `elements` | `ParsedElement[]` | Array of detected UI elements |
+| `annotatedImageUrl` | `string` | URL of the annotated screenshot with bounding boxes |
+| `imageWidth` | `number` | Width of the analyzed screenshot |
+| `imageHeight` | `number` | Height of the analyzed screenshot |
+
+### ParsedElement
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `index` | `number` | Element index |
+| `type` | `string` | Element type (e.g. `"text"`, `"icon"`, `"button"`) |
+| `content` | `string` | Text content or description of the element |
+| `interactivity` | `string` | Interactivity level (e.g. `"clickable"`, `"non-interactive"`) |
+| `bbox` | `object` | Bounding box in pixel coordinates `{x0, y0, x1, y1}` |
+| `boundingBox` | `object` | Bounding box as `{left, top, width, height}` |
+
+## Examples
+
+### Get All Elements on Screen
+
+```javascript
+const result = await testdriver.parse();
+console.log(`Found ${result.elements.length} elements`);
+
+result.elements.forEach((el, i) => {
+  console.log(`${i + 1}. [${el.type}] "${el.content}" (${el.interactivity})`);
+});
+```
+
+### Find Clickable Elements
+
+```javascript
+const result = await testdriver.parse();
+
+const clickable = result.elements.filter(e => e.interactivity === 'clickable');
+console.log(`Found ${clickable.length} clickable elements`);
+
+clickable.forEach(el => {
+  console.log(`- "${el.content}" at (${el.bbox.x0}, ${el.bbox.y0})`);
+});
+```
+
+### Find and Click an Element by Content
+
+```javascript
+const result = await testdriver.parse();
+
+// Find a "Submit" button
+const submitBtn = result.elements.find(e => 
+  e.content.toLowerCase().includes('submit') && e.interactivity === 'clickable'
+);
+
+if (submitBtn) {
+  // Calculate center of the bounding box
+  const x = Math.round((submitBtn.bbox.x0 + submitBtn.bbox.x1) / 2);
+  const y = Math.round((submitBtn.bbox.y0 + submitBtn.bbox.y1) / 2);
+  
+  await testdriver.click({ x, y });
+}
+```
+
+### Filter by Element Type
+
+```javascript
+const result = await testdriver.parse();
+
+// Get all text elements
+const textElements = result.elements.filter(e => e.type === 'text');
+textElements.forEach(e => console.log(`Text: "${e.content}"`));
+
+// Get all icons
+const icons = result.elements.filter(e => e.type === 'icon');
+console.log(`Found ${icons.length} icons`);
+
+// Get all buttons
+const buttons = result.elements.filter(e => e.type === 'button');
+console.log(`Found ${buttons.length} buttons`);
+```
+
+### Build Custom Assertions
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+
+describe("Login Page", () => {
+  it("should have expected form elements", async (context) => {
+    const testdriver = TestDriver(context);
+    
+    await testdriver.provision.chrome({
+      url: 'https://myapp.com/login',
+    });
+
+    const result = await testdriver.parse();
+    
+    // Assert expected elements exist
+    const textContent = result.elements.map(e => e.content.toLowerCase());
+    expect(textContent).toContain('email');
+    expect(textContent).toContain('password');
+    
+    // Assert there are clickable elements
+    const clickable = result.elements.filter(e => e.interactivity === 'clickable');
+    expect(clickable.length).toBeGreaterThan(0);
+  });
+});
+```
+
+### Use Bounding Box Coordinates
+
+```javascript
+const result = await testdriver.parse();
+
+result.elements.forEach(el => {
+  // Pixel coordinates
+  console.log(`Element "${el.content}":`);
+  console.log(`  bbox: (${el.bbox.x0}, ${el.bbox.y0}) to (${el.bbox.x1}, ${el.bbox.y1})`);
+  console.log(`  size: ${el.boundingBox.width}x${el.boundingBox.height}`);
+  console.log(`  position: left=${el.boundingBox.left}, top=${el.boundingBox.top}`);
+});
+```
+
+### View Annotated Screenshot
+
+```javascript
+const result = await testdriver.parse();
+
+// The annotated image shows all detected elements with bounding boxes
+console.log('Annotated screenshot:', result.annotatedImageUrl);
+console.log(`Image dimensions: ${result.imageWidth}x${result.imageHeight}`);
+```
+
+## How It Works
+
+1. TestDriver captures a screenshot of the current screen
+2. The image is sent to the TestDriver API
+3. OmniParser v2 analyzes the image to detect all UI elements
+4. Each element is classified by type (text, icon, button, etc.) and interactivity
+5. Bounding box coordinates are returned in pixel coordinates matching the screen resolution
+
+<Note>
+  OmniParser detects elements visually — it works with any UI framework, native apps, and even non-standard interfaces. It does not rely on DOM or accessibility trees.
+</Note>
+
+## Best Practices
+
+<AccordionGroup>
+  <Accordion title="Use find() for targeting specific elements">
+    For locating and interacting with a specific element, prefer `find()` which uses AI vision. Use `parse()` when you need a complete inventory of all elements on screen.
+    
+    ```javascript
+    // Prefer this for clicking a specific element
+    await testdriver.find("Submit button").click();
+    
+    // Use parse() for full UI analysis
+    const result = await testdriver.parse();
+    const allButtons = result.elements.filter(e => e.type === 'button');
+    ```
+  </Accordion>
+  
+  <Accordion title="Filter by interactivity">
+    Use the `interactivity` field to distinguish between clickable and non-interactive elements.
+    
+    ```javascript
+    const result = await testdriver.parse();
+    const interactive = result.elements.filter(e => e.interactivity === 'clickable');
+    const static_ = result.elements.filter(e => e.interactivity === 'non-interactive');
+    ```
+  </Accordion>
+
+  <Accordion title="Wait for content to load">
+    If elements aren't being detected, the page may not be fully loaded. Add a wait first.
+    
+    ```javascript
+    // Wait for page to stabilize
+    await testdriver.wait(2000);
+    
+    // Then parse
+    const result = await testdriver.parse();
+    ```
+  </Accordion>
+  
+  <Accordion title="Use the annotated image for debugging">
+    The `annotatedImageUrl` provides a visual overlay showing all detected elements with their bounding boxes — great for debugging.
+    
+    ```javascript
+    const result = await testdriver.parse();
+    console.log('View annotated screenshot:', result.annotatedImageUrl);
+    ```
+  </Accordion>
+</AccordionGroup>
+
+## Related
+
+- [find()](/v7/find) - AI-powered element location
+- [assert()](/v7/assert) - Make AI-powered assertions about screen state
+- [screenshot()](/v7/screenshot) - Capture screenshots
+- [Elements Reference](/v7/elements) - Complete Element API

package/ai/skills/testdriver:performing-actions/SKILL.md

@@ -0,0 +1,53 @@
+---
+name: testdriver:performing-actions
+description: Click, type, hover, scroll and more with TestDriver
+---
+<!-- Generated from performing-actions.mdx. DO NOT EDIT. -->
+
+## Performing Actions
+
+TestDriver provides a variety of actions you can perform, like [clicking](/v7/click), [typing](/v7/type), [hovering](/v7/hover), and [scrolling](/v7/scroll). For a full list, see the [API Reference](/v7/click).
+
+```javascript
+// Clicking
+await testdriver.find('submit button').click();
+await testdriver.find('file item').doubleClick();
+await testdriver.find('text area').rightClick();
+
+// Typing
+await testdriver.find('email input').type('user@example.com');
+await testdriver.find('password input').type('secret', { secret: true });
+
+// Keyboard shortcuts
+await testdriver.pressKeys(['enter']);
+await testdriver.pressKeys(['ctrl', 'c']);
+
+// Hovering
+await testdriver.find('dropdown menu').hover();
+
+// Scrolling
+await testdriver.scroll('down', 500);
+
+// Waiting
+await testdriver.wait(2000); // Wait 2 seconds for animation/state change
+
+// Extracting information from screen
+const price = await testdriver.extract('the total price');
+const orderNumber = await testdriver.extract('the order confirmation number');
+```
+
+## Chaining Actions
+
+TestDriver supports method chaining for cleaner code:
+
+```javascript
+// Chain find() with actions
+const button = await testdriver.find('submit button').click();
+```
+
+Or save element reference for later use:
+
+```javascript
+const button = await testdriver.find('submit button');
+await button.click();
+```

package/ai/skills/testdriver:press-keys/SKILL.md

@@ -0,0 +1,348 @@
+---
+name: testdriver:press-keys
+description: Press keyboard keys and shortcuts
+---
+<!-- Generated from press-keys.mdx. DO NOT EDIT. -->
+
+## Overview
+
+Press one or more keyboard keys simultaneously, useful for keyboard shortcuts, navigation, and special keys.
+
+## Syntax
+
+```javascript
+await testdriver.pressKeys(keys)
+```
+
+## Parameters
+
+<ParamField path="keys" type="Array&lt;string&gt;" required>
+  Array of keys to press simultaneously
+</ParamField>
+
+## Returns
+
+`Promise<void>`
+
+## Common Keys
+
+### Special Keys
+- `'enter'`, `'tab'`, `'escape'`, `'backspace'`, `'delete'`
+- `'space'`, `'up'`, `'down'`, `'left'`, `'right'`
+- `'home'`, `'end'`, `'pageup'`, `'pagedown'`
+
+### Modifier Keys
+- `'ctrl'`, `'alt'`, `'shift'`
+- `'command'` (macOS), `'win'` (Windows)
+- `'ctrlleft'`, `'ctrlright'`, `'shiftleft'`, `'shiftright'`
+
+### Function Keys
+- `'f1'` through `'f24'`
+
+## Examples
+
+### Navigation
+
+```javascript
+// Tab to next field
+await testdriver.pressKeys(['tab']);
+
+// Shift+Tab to previous field
+await testdriver.pressKeys(['shift', 'tab']);
+
+// Arrow keys
+await testdriver.pressKeys(['down']);
+await testdriver.pressKeys(['up']);
+await testdriver.pressKeys(['left']);
+await testdriver.pressKeys(['right']);
+
+// Home/End
+await testdriver.pressKeys(['home']);  // Start of line
+await testdriver.pressKeys(['end']);   // End of line
+
+// Page navigation
+await testdriver.pressKeys(['pagedown']);
+await testdriver.pressKeys(['pageup']);
+```
+
+### Keyboard Shortcuts
+
+```javascript
+// Copy (Ctrl+C / Cmd+C)
+await testdriver.pressKeys(['ctrl', 'c']);
+
+// Paste (Ctrl+V / Cmd+V)
+await testdriver.pressKeys(['ctrl', 'v']);
+
+// Save (Ctrl+S)
+await testdriver.pressKeys(['ctrl', 's']);
+
+// Select All (Ctrl+A)
+await testdriver.pressKeys(['ctrl', 'a']);
+
+// Undo (Ctrl+Z)
+await testdriver.pressKeys(['ctrl', 'z']);
+
+// Redo (Ctrl+Y)
+await testdriver.pressKeys(['ctrl', 'y']);
+
+// Find (Ctrl+F)
+await testdriver.pressKeys(['ctrl', 'f']);
+
+// New tab (Ctrl+T)
+await testdriver.pressKeys(['ctrl', 't']);
+
+// Close tab (Ctrl+W)
+await testdriver.pressKeys(['ctrl', 'w']);
+
+// Refresh (F5 or Ctrl+R)
+await testdriver.pressKeys(['f5']);
+await testdriver.pressKeys(['ctrl', 'r']);
+```
+
+### System Shortcuts
+
+```javascript
+// Alt+Tab (Windows - switch apps)
+await testdriver.pressKeys(['alt', 'tab']);
+
+// Alt+F4 (Windows - close window)
+await testdriver.pressKeys(['alt', 'f4']);
+
+// Win+D (Windows - show desktop)
+await testdriver.pressKeys(['winleft', 'd']);
+
+// Win+L (Windows - lock screen)
+await testdriver.pressKeys(['winleft', 'l']);
+
+// Cmd+Tab (macOS - switch apps)
+await testdriver.pressKeys(['command', 'tab']);
+
+// Cmd+Q (macOS - quit app)
+await testdriver.pressKeys(['command', 'q']);
+```
+
+### Form Submission
+
+```javascript
+// Submit form
+await testdriver.pressKeys(['enter']);
+
+// Cancel/Close
+await testdriver.pressKeys(['escape']);
+
+// Check checkbox
+await testdriver.pressKeys(['space']);
+```
+
+### Text Editing
+
+```javascript
+// Delete selected text
+await testdriver.pressKeys(['delete']);
+
+// Backspace
+await testdriver.pressKeys(['backspace']);
+
+// Select all and delete
+await testdriver.pressKeys(['ctrl', 'a']);
+await testdriver.pressKeys(['delete']);
+
+// Cut text
+await testdriver.pressKeys(['ctrl', 'x']);
+```
+
+## Best Practices
+
+<Check>
+  **Wait after shortcuts**
+  
+  Some keyboard shortcuts trigger animations or navigation:
+  
+  ```javascript
+  await testdriver.pressKeys(['ctrl', 't']); // New tab
+  await new Promise(r => setTimeout(r, 500)); // Wait for tab
+  await testdriver.pressKeys(['ctrl', 'l']); // Focus URL bar
+  ```
+</Check>
+
+<Check>
+  **Use Tab for form navigation**
+  
+  Tab is more reliable than clicking multiple fields:
+  
+  ```javascript
+  const firstField = await testdriver.find('email input');
+  await firstField.click();
+  await testdriver.type('user@example.com');
+  
+  await testdriver.pressKeys(['tab']);
+  await testdriver.type('password123');
+  
+  await testdriver.pressKeys(['tab']);
+  await testdriver.pressKeys(['enter']); // Submit
+  ```
+</Check>
+
+<Warning>
+  **Platform-specific keys**
+  
+  Use the appropriate modifier key for the platform:
+  - Windows/Linux: `'ctrl'`
+  - macOS: `'command'`
+  
+  ```javascript
+  // For cross-platform, you might need to detect OS
+  const modKey = process.platform === 'darwin' ? 'command' : 'ctrl';
+  await testdriver.pressKeys([modKey, 'c']); // Copy
+  ```
+</Warning>
+
+## Use Cases
+
+<AccordionGroup>
+  <Accordion title="Form Navigation">
+    ```javascript
+    // Fill form using Tab
+    const firstField = await testdriver.find('name field');
+    await firstField.click();
+    await testdriver.type('John Doe');
+    
+    await testdriver.pressKeys(['tab']);
+    await testdriver.type('john@example.com');
+    
+    await testdriver.pressKeys(['tab']);
+    await testdriver.type('555-0123');
+    
+    await testdriver.pressKeys(['tab']);
+    await testdriver.pressKeys(['enter']); // Submit
+    ```
+  </Accordion>
+  
+  <Accordion title="Text Manipulation">
+    ```javascript
+    const textArea = await testdriver.find('comment textarea');
+    await textArea.click();
+    
+    // Select all existing text
+    await testdriver.pressKeys(['ctrl', 'a']);
+    
+    // Copy it
+    await testdriver.pressKeys(['ctrl', 'c']);
+    
+    // Type new text
+    await testdriver.type('New comment');
+    
+    // Undo if needed
+    await testdriver.pressKeys(['ctrl', 'z']);
+    ```
+  </Accordion>
+  
+  <Accordion title="Browser Navigation">
+    ```javascript
+    // Open new tab
+    await testdriver.pressKeys(['ctrl', 't']);
+    await new Promise(r => setTimeout(r, 500));
+    
+    // Focus address bar
+    await testdriver.pressKeys(['ctrl', 'l']);
+    await testdriver.type('https://example.com');
+    await testdriver.pressKeys(['enter']);
+    
+    // Refresh page
+    await testdriver.pressKeys(['f5']);
+    
+    // Close tab
+    await testdriver.pressKeys(['ctrl', 'w']);
+    ```
+  </Accordion>
+  
+  <Accordion title="Application Shortcuts">
+    ```javascript
+    // Save document
+    await testdriver.pressKeys(['ctrl', 's']);
+    
+    // Print
+    await testdriver.pressKeys(['ctrl', 'p']);
+    
+    // Find in page
+    await testdriver.pressKeys(['ctrl', 'f']);
+    await testdriver.type('search term');
+    await testdriver.pressKeys(['escape']); // Close find
+    ```
+  </Accordion>
+</AccordionGroup>
+
+## Complete Example
+
+```javascript
+import { beforeAll, afterAll, describe, it } from 'vitest';
+import TestDriver from 'testdriverai';
+
+describe('Keyboard Navigation', () => {
+  let testdriver;
+
+  beforeAll(async () => {
+    client = new TestDriver(process.env.TD_API_KEY);
+    await testdriver.auth();
+    await testdriver.connect();
+  });
+
+  afterAll(async () => {
+    await testdriver.disconnect();
+  });
+
+  it('should navigate form with keyboard', async () => {
+    await testdriver.focusApplication('Google Chrome');
+    
+    // Find first field
+    const emailField = await testdriver.find('email input');
+    await emailField.click();
+    await testdriver.type('user@example.com');
+    
+    // Tab through fields
+    await testdriver.pressKeys(['tab']);
+    await testdriver.type('John');
+    
+    await testdriver.pressKeys(['tab']);
+    await testdriver.type('Doe');
+    
+    await testdriver.pressKeys(['tab']);
+    await testdriver.type('password123');
+    
+    // Submit with Enter
+    await testdriver.pressKeys(['tab']);
+    await testdriver.pressKeys(['enter']);
+    
+    await testdriver.assert('form submitted successfully');
+  });
+
+  it('should use keyboard shortcuts', async () => {
+    // Open new browser tab
+    await testdriver.pressKeys(['ctrl', 't']);
+    await new Promise(r => setTimeout(r, 500));
+    
+    // Focus address bar
+    await testdriver.pressKeys(['ctrl', 'l']);
+    await testdriver.type('https://example.com');
+    await testdriver.pressKeys(['enter']);
+    
+    await new Promise(r => setTimeout(r, 2000));
+    
+    // Select all page content
+    await testdriver.pressKeys(['ctrl', 'a']);
+    
+    // Copy
+    await testdriver.pressKeys(['ctrl', 'c']);
+    
+    // Refresh page
+    await testdriver.pressKeys(['f5']);
+  });
+});
+```
+
+## Related Methods
+
+- [`type()`](/v7/type) - Type text
+- [`click()`](/v7/click) - Click elements
+- [`scroll()`](/v7/scroll) - Scroll pages