@testdriverai/agent 7.8.0-test.38

package/ai/skills/testdriver-mouse-up/SKILL.md

@@ -0,0 +1,164 @@
+---
+name: testdriver:mouse-up
+description: Release the mouse button
+---
+<!-- Generated from mouse-up.mdx. DO NOT EDIT. -->
+
+## Overview
+
+The `mouseUp()` method releases the mouse button, completing a drag operation or custom mouse gesture that was started with [`mouseDown()`](/v7/mouse-down). You can call it without parameters to release at the current mouse position.
+
+## Syntax
+
+```javascript
+// Release mouse button at current position
+await ai.mouseUp();
+```
+
+## Parameters
+
+None. The mouse button is released at the current cursor position.
+
+## Returns
+
+Returns a `Promise<void>` that resolves when the mouse button is released.
+
+## Examples
+
+### Complete Drag and Drop
+
+```javascript
+// Start dragging
+await ai.mouseDown('file to drag');
+
+// Move to drop location
+await ai.hover('target folder');
+
+// Complete the drop
+await ai.mouseUp();
+```
+
+### Selecting Multiple Files
+
+```javascript
+import { test } from 'vitest';
+import { vscode } from '@testdriver/sdk';
+
+test('selects range of files', async () => {
+  const { ai } = await vscode();
+  
+  // Click first file
+  await ai.click('first-file.js in explorer');
+  
+  // Hold shift and click last file
+  await ai.pressKeys('Shift');
+  await ai.mouseDown('last-file.js in explorer');
+  await ai.mouseUp();
+  
+  // Verify multiple files selected
+  const selectedCount = await ai.find('status bar showing file count');
+  expect(selectedCount.text).toContain('5 files');
+});
+```
+
+### Drawing Application
+
+```javascript
+test('draws a shape', async () => {
+  const { ai } = await chrome('https://drawing-app.example.com');
+  
+  // Select pencil tool
+  await ai.click('pencil tool');
+  
+  // Draw a line
+  await ai.mouseDown('canvas near top-left');
+  await ai.hover('canvas center');
+  await ai.hover('canvas bottom-right');
+  await ai.mouseUp();
+  
+  // Verify drawing exists
+  const strokes = await ai.exec('canvas.getContext("2d").getImageData(0,0,100,100)');
+  expect(strokes).toBeTruthy();
+});
+```
+
+### Resizing Window Panels
+
+```javascript
+test('resizes editor panel', async () => {
+  const { ai } = await vscode();
+  
+  // Grab the divider
+  await ai.mouseDown('panel resize divider');
+  
+  // Drag to new position
+  await ai.hover('position 400 pixels from left');
+  
+  // Release to complete resize
+  await ai.mouseUp();
+  
+  // Verify new panel size
+  const panel = await ai.find('editor panel');
+  expect(panel.width).toBeGreaterThan(350);
+});
+```
+
+### Drag to Reorder List Items
+
+```javascript
+import { test } from 'vitest';
+import { chrome } from '@testdriver/sdk';
+
+test('reorders tasks in list', async () => {
+  const { ai } = await chrome('https://todo-app.example.com');
+  
+  // Start dragging first task
+  await ai.mouseDown('drag handle on first task');
+  
+  // Move down to third position
+  await ai.hover('third task position');
+  
+  // Drop the task
+  await ai.mouseUp();
+  
+  // Verify new order
+  const thirdTask = await ai.find('third task in list');
+  expect(thirdTask.text).toContain('Original first task');
+});
+```
+
+### Text Selection with Mouse
+
+```javascript
+test('selects text with mouse drag', async () => {
+  const { ai } = await chrome('https://document.example.com');
+  
+  // Start selection at beginning of word
+  await ai.mouseDown('start of "TestDriver" word');
+  
+  // Drag to end of word
+  await ai.hover('end of "TestDriver" word');
+  
+  // Complete selection
+  await ai.mouseUp();
+  
+  // Verify selection
+  const selection = await ai.exec('window.getSelection().toString()');
+  expect(selection).toBe('TestDriver');
+});
+```
+
+## Important Notes
+
+- `mouseUp()` must be preceded by [`mouseDown()`](/v7/mouse-down) to have an effect
+- Releases the button at the current cursor position
+- Completes any drag or selection operation that was in progress
+- For simple clicks, use [`click()`](/v7/click) instead of mouseDown/mouseUp pair
+
+## Related Methods
+
+- [`mouseDown()`](/v7/mouse-down) - Press mouse button without releasing
+- [`hover()`](/v7/hover) - Move mouse to element
+- [`click()`](/v7/click) - Complete click (mouseDown + mouseUp)
+- [`doubleClick()`](/v7/double-click) - Double-click on element
+- [`rightClick()`](/v7/right-click) - Right-click for context menu

package/ai/skills/testdriver-parse/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: testdriver:parse
+description: Detect all UI elements on screen using OmniParser
+---
+<!-- Generated from parse.mdx. DO NOT EDIT. -->
+
+## Overview
+
+Parse the current screen using OmniParser v2 to detect all visible UI elements. Returns structured data including element types, text content, interactivity levels, and bounding box coordinates.
+
+This method analyzes the entire screen and returns every detected element. It's useful for:
+- Understanding the full UI layout of a screen
+- Finding all clickable or interactive elements
+- Building custom element-based logic
+- Debugging what elements TestDriver can detect
+- Accessibility auditing
+
+<Note>
+  **Availability**: `parse()` requires an enterprise or self-hosted plan. It uses OmniParser v2 server-side for element detection.
+</Note>
+
+## Syntax
+
+```javascript
+const result = await testdriver.parse()
+```
+
+## Parameters
+
+None.
+
+## Returns
+
+`Promise<ParseResult>` - Object containing detected UI elements
+
+### ParseResult
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `elements` | `ParsedElement[]` | Array of detected UI elements |
+| `annotatedImageUrl` | `string` | URL of the annotated screenshot with bounding boxes |
+| `imageWidth` | `number` | Width of the analyzed screenshot |
+| `imageHeight` | `number` | Height of the analyzed screenshot |
+
+### ParsedElement
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `index` | `number` | Element index |
+| `type` | `string` | Element type (e.g. `"text"`, `"icon"`, `"button"`) |
+| `content` | `string` | Text content or description of the element |
+| `interactivity` | `string` | Interactivity level (e.g. `"clickable"`, `"non-interactive"`) |
+| `bbox` | `object` | Bounding box in pixel coordinates `{x0, y0, x1, y1}` |
+| `boundingBox` | `object` | Bounding box as `{left, top, width, height}` |
+
+## Examples
+
+### Get All Elements on Screen
+
+```javascript
+const result = await testdriver.parse();
+console.log(`Found ${result.elements.length} elements`);
+
+result.elements.forEach((el, i) => {
+  console.log(`${i + 1}. [${el.type}] "${el.content}" (${el.interactivity})`);
+});
+```
+
+### Find Clickable Elements
+
+```javascript
+const result = await testdriver.parse();
+
+const clickable = result.elements.filter(e => e.interactivity === 'clickable');
+console.log(`Found ${clickable.length} clickable elements`);
+
+clickable.forEach(el => {
+  console.log(`- "${el.content}" at (${el.bbox.x0}, ${el.bbox.y0})`);
+});
+```
+
+### Find and Click an Element by Content
+
+```javascript
+const result = await testdriver.parse();
+
+// Find a "Submit" button
+const submitBtn = result.elements.find(e => 
+  e.content.toLowerCase().includes('submit') && e.interactivity === 'clickable'
+);
+
+if (submitBtn) {
+  // Calculate center of the bounding box
+  const x = Math.round((submitBtn.bbox.x0 + submitBtn.bbox.x1) / 2);
+  const y = Math.round((submitBtn.bbox.y0 + submitBtn.bbox.y1) / 2);
+  
+  await testdriver.click({ x, y });
+}
+```
+
+### Filter by Element Type
+
+```javascript
+const result = await testdriver.parse();
+
+// Get all text elements
+const textElements = result.elements.filter(e => e.type === 'text');
+textElements.forEach(e => console.log(`Text: "${e.content}"`));
+
+// Get all icons
+const icons = result.elements.filter(e => e.type === 'icon');
+console.log(`Found ${icons.length} icons`);
+
+// Get all buttons
+const buttons = result.elements.filter(e => e.type === 'button');
+console.log(`Found ${buttons.length} buttons`);
+```
+
+### Build Custom Assertions
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("Login Page", () => {
+  it("should have expected form elements", async (context) => {
+    const testdriver = TestDriver(context);
+    
+    await testdriver.provision.chrome({
+      url: 'https://myapp.com/login',
+    });
+
+    const result = await testdriver.parse();
+    
+    // Assert expected elements exist
+    const textContent = result.elements.map(e => e.content.toLowerCase());
+    expect(textContent).toContain('email');
+    expect(textContent).toContain('password');
+    
+    // Assert there are clickable elements
+    const clickable = result.elements.filter(e => e.interactivity === 'clickable');
+    expect(clickable.length).toBeGreaterThan(0);
+  });
+});
+```
+
+### Use Bounding Box Coordinates
+
+```javascript
+const result = await testdriver.parse();
+
+result.elements.forEach(el => {
+  // Pixel coordinates
+  console.log(`Element "${el.content}":`);
+  console.log(`  bbox: (${el.bbox.x0}, ${el.bbox.y0}) to (${el.bbox.x1}, ${el.bbox.y1})`);
+  console.log(`  size: ${el.boundingBox.width}x${el.boundingBox.height}`);
+  console.log(`  position: left=${el.boundingBox.left}, top=${el.boundingBox.top}`);
+});
+```
+
+### View Annotated Screenshot
+
+```javascript
+const result = await testdriver.parse();
+
+// The annotated image shows all detected elements with bounding boxes
+console.log('Annotated screenshot:', result.annotatedImageUrl);
+console.log(`Image dimensions: ${result.imageWidth}x${result.imageHeight}`);
+```
+
+## How It Works
+
+1. TestDriver captures a screenshot of the current screen
+2. The image is sent to the TestDriver API
+3. OmniParser v2 analyzes the image to detect all UI elements
+4. Each element is classified by type (text, icon, button, etc.) and interactivity
+5. Bounding box coordinates are returned in pixel coordinates matching the screen resolution
+
+<Note>
+  OmniParser detects elements visually — it works with any UI framework, native apps, and even non-standard interfaces. It does not rely on DOM or accessibility trees.
+</Note>
+
+## Best Practices
+
+<AccordionGroup>
+  <Accordion title="Use find() for targeting specific elements">
+    For locating and interacting with a specific element, prefer `find()` which uses AI vision. Use `parse()` when you need a complete inventory of all elements on screen.
+    
+    ```javascript
+    // Prefer this for clicking a specific element
+    await testdriver.find("Submit button").click();
+    
+    // Use parse() for full UI analysis
+    const result = await testdriver.parse();
+    const allButtons = result.elements.filter(e => e.type === 'button');
+    ```
+  </Accordion>
+  
+  <Accordion title="Filter by interactivity">
+    Use the `interactivity` field to distinguish between clickable and non-interactive elements.
+    
+    ```javascript
+    const result = await testdriver.parse();
+    const interactive = result.elements.filter(e => e.interactivity === 'clickable');
+    const static_ = result.elements.filter(e => e.interactivity === 'non-interactive');
+    ```
+  </Accordion>
+
+  <Accordion title="Wait for content to load">
+    If elements aren't being detected, the page may not be fully loaded. Add a wait first.
+    
+    ```javascript
+    // Wait for page to stabilize
+    await testdriver.wait(2000);
+    
+    // Then parse
+    const result = await testdriver.parse();
+    ```
+  </Accordion>
+  
+  <Accordion title="Use the annotated image for debugging">
+    The `annotatedImageUrl` provides a visual overlay showing all detected elements with their bounding boxes — great for debugging.
+    
+    ```javascript
+    const result = await testdriver.parse();
+    console.log('View annotated screenshot:', result.annotatedImageUrl);
+    ```
+  </Accordion>
+</AccordionGroup>
+
+## Related
+
+- [find()](/v7/find) - AI-powered element location
+- [assert()](/v7/assert) - Make AI-powered assertions about screen state
+- [screenshot()](/v7/screenshot) - Capture screenshots
+- [Elements Reference](/v7/elements) - Complete Element API

package/ai/skills/testdriver-performing-actions/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: testdriver:performing-actions
+description: Click, type, hover, scroll and more with TestDriver
+---
+<!-- Generated from performing-actions.mdx. DO NOT EDIT. -->
+
+## Performing Actions
+
+TestDriver provides a variety of actions you can perform, like [clicking](/v7/click), [typing](/v7/type), [hovering](/v7/hover), and [scrolling](/v7/scroll). For a full list, see the [API Reference](/v7/click).
+
+```javascript
+// Clicking
+await testdriver.find('submit button').click();
+await testdriver.find('file item').doubleClick();
+await testdriver.find('text area').rightClick();
+
+// Typing
+await testdriver.find('email input').type('user@example.com');
+await testdriver.find('password input').type('secret', { secret: true });
+
+// Keyboard shortcuts
+await testdriver.pressKeys(['enter']);
+await testdriver.pressKeys(['ctrl', 'c']);
+
+// Hovering
+await testdriver.find('dropdown menu').hover();
+
+// Scrolling
+await testdriver.scroll('down', 500);
+await testdriver.scrollUntilText('Footer content');
+
+// Waiting
+await testdriver.wait(2000); // Wait 2 seconds for animation/state change
+
+// Extracting information from screen
+const price = await testdriver.extract('the total price');
+const orderNumber = await testdriver.extract('the order confirmation number');
+```
+
+## Chaining Actions
+
+TestDriver supports method chaining for cleaner code:
+
+```javascript
+// Chain find() with actions
+const button = await testdriver.find('submit button').click();
+```
+
+Or save element reference for later use:
+
+```javascript
+const button = await testdriver.find('submit button');
+await button.click();
+```