testdriverai 7.9.32-test → 7.9.34-test

package/ai/skills/testdriver:running-tests/SKILL.md

@@ -0,0 +1,185 @@
+---
+name: testdriver:running-tests
+description: Run TestDriver tests with Vitest test runner
+---
+<!-- Generated from running-tests.mdx. DO NOT EDIT. -->
+
+Learn how to run TestDriver tests efficiently with Vitest's powerful test runner.
+
+## Running Tests
+
+TestDriver works with Vitest's powerful test runner.
+
+<Info>
+  Install Vitest globally for best results: `npm install vitest -g`
+</Info>
+
+### Run All Tests
+
+```bash
+vitest run
+```
+
+Executes all test files in your project once and exits. Vitest automatically discovers files matching patterns like `*.test.js`, `*.test.mjs`, or `*.spec.js`.
+
+### Run with Coverage
+
+```bash
+vitest run --coverage
+```
+
+Generates a code coverage report showing which lines of your source code were executed during tests. Coverage helps identify untested code paths. Results are displayed in the terminal and saved to a `coverage/` directory.
+
+<Info>
+  Coverage requires the `@vitest/coverage-v8` package. Install it with `npm install -D @vitest/coverage-v8`.
+</Info>
+
+### Run Specific Tests
+
+```bash
+vitest run login.test.js
+```
+
+Runs only the specified test file. Useful when debugging a single test or working on a specific feature.
+
+```bash
+vitest run login.test.js checkout.test.js
+```
+
+Runs multiple specific test files. List as many files as needed, separated by spaces.
+
+### Filter Tests by Name
+
+```bash
+vitest run --grep "login"
+```
+
+The `--grep` flag filters tests by their name (the string passed to `it()` or `test()`). Only tests whose names match the pattern will run. Supports regex patterns for complex matching.
+
+### Run Tests in a Folder
+
+```bash
+vitest run tests/e2e/
+```
+
+Runs all test files within a specific directory. Great for organizing tests by type (unit, integration, e2e) and running them separately.
+
+## Parallel Execution
+
+TestDriver runs each test in its own cloud sandbox, enabling true parallel execution. Run your entire test suite in minutes instead of hours.
+
+### Control Concurrency
+
+```bash
+vitest run --maxConcurrency=5
+```
+
+The `--maxConcurrency` flag limits how many tests run simultaneously. This should match your TestDriver license slots to avoid failures from exhausted slots.
+
+### Thread Configuration
+
+```bash
+vitest run --pool=threads --minThreads=2 --maxThreads=8
+```
+
+Fine-tune thread allocation for optimal performance:
+- `--pool=threads` — Uses worker threads for test isolation
+- `--minThreads` — Minimum number of threads to keep alive (reduces startup overhead)
+- `--maxThreads` — Maximum threads to spawn (limits resource usage)
+
+### License Slots
+
+Your TestDriver plan includes a set number of **license slots** that determine how many tests can run simultaneously. Each running test occupies one slot—when the test completes and the sandbox is destroyed, the slot is immediately available for the next test.
+
+<Info>
+  View your available slots at [console.testdriver.ai](https://console.testdriver.ai). Upgrade anytime to increase parallelization.
+</Info>
+
+### Configuring Concurrency
+
+Set `maxConcurrency` in your Vitest config to match your license slot limit:
+
+```javascript vitest.config.mjs
+import { defineConfig } from 'vitest/config';
+import TestDriver from 'testdriverai/vitest';
+
+export default defineConfig({
+  test: {
+    testTimeout: 900000,
+    hookTimeout: 900000,
+    maxConcurrency: 5, // Match your license slot limit
+    reporters: ['default', TestDriver()],
+    setupFiles: ['testdriverai/vitest/setup'],
+  },
+});
+```
+
+<Warning>
+  Setting `maxConcurrency` higher than your license slots will cause tests to fail when slots are exhausted. Always match this value to your plan's limit.
+</Warning>
+
+### Why Parallelization Matters
+
+| Test Suite | Sequential (1 slot) | Parallel (5 slots) | Parallel (10 slots) |
+|------------|--------------------|--------------------|---------------------|
+| 10 tests @ 2min each | 20 min | 4 min | 2 min |
+| 50 tests @ 2min each | 100 min | 20 min | 10 min |
+| 100 tests @ 2min each | 200 min | 40 min | 20 min |
+
+<Tip>
+  **Pro tip:** Upgrading your plan doesn't just increase speed—it enables faster CI/CD feedback loops, letting your team ship with confidence.
+</Tip>
+
+<Card
+  title="View Plans & Pricing"
+  icon="credit-card"
+  href="/v7/hosted"
+>
+  Compare plans and find the right level of parallelization for your team.
+</Card>
+
+## Vitest UI
+
+Use Vitest UI for interactive debugging:
+
+```bash
+vitest --ui
+```
+
+The `--ui` flag launches a web-based interface for managing your test suite. Unlike `vitest run`, this starts in watch mode by default.
+
+Open http://localhost:51204 to see:
+- **Test file tree** — Browse and navigate your test structure
+- **Test status and duration** — See pass/fail states and timing at a glance
+- **Console output** — View logs and errors inline with each test
+- **Re-run individual tests** — Click to re-execute specific tests without restarting
+- **Filter and search** — Quickly find tests by name or status
+
+<Tip>
+  Combine with `--open` to automatically open the UI in your browser: `vitest --ui --open`
+</Tip>
+
+
+## Test Reports
+
+After running tests, view detailed reports and video replays at [console.testdriver.ai](https://console.testdriver.ai).
+
+Reports include:
+- **Video replays** - Watch exactly what happened during each test
+- **Screenshots** - See the state at each step
+- **Timing breakdown** - Identify slow operations
+- **Error details** - Debug failures with full context
+
+```bash
+$ vitest run
+
+ ✓ login.test.js (2) 18.4s
+   ✓ user can login 12.3s
+   ✓ shows error for invalid credentials 6.1s
+
+📹 View reports at: https://console.testdriver.ai
+```
+
+<Tip>
+  Bookmark your team's dashboard at [console.testdriver.ai](https://console.testdriver.ai) for quick access to test history and analytics.
+</Tip>

package/ai/skills/testdriver:screenshot/SKILL.md

@@ -0,0 +1,248 @@
+---
+name: testdriver:screenshot
+description: Capture and save screenshots during test execution
+---
+<!-- Generated from screenshot.mdx. DO NOT EDIT. -->
+
+## Overview
+
+Capture a screenshot of the current screen and automatically save it to a local file. Screenshots are organized by test file for easy debugging and review.
+
+<Note>
+  **Automatic Screenshots**: TestDriver can automatically capture screenshots before and after every command (click, type, find, etc.). These are saved with descriptive filenames like `001-click-before-L42-submit-button.png` that include the line number from your test file. Enable this with `autoScreenshots: true` in your TestDriver options.
+</Note>
+
+## Syntax
+
+```javascript
+const filePath = await testdriver.screenshot(filename)
+```
+
+## Parameters
+
+<ParamField path="filename" type="string" optional>
+  Custom filename for the screenshot (without .png extension). If not provided, a timestamp-based filename is generated automatically.
+</ParamField>
+
+## Returns
+
+`Promise<string>` - The absolute file path where the screenshot was saved
+
+## File Organization
+
+Screenshots are automatically saved to `.testdriver/screenshots/<test-file-name>/` in your project root:
+
+```
+.testdriver/
+  screenshots/
+    login.test/
+      001-find-before-L15-email-input.png     # Auto: before find()
+      002-find-after-L15-email-input.png      # Auto: after find()
+      003-click-before-L16-email-input.png    # Auto: before click()
+      004-click-after-L16-email-input.png     # Auto: after click()
+      005-type-before-L17-userexamplecom.png  # Auto: before type()
+      006-type-after-L17-userexamplecom.png   # Auto: after type()
+      custom-screenshot.png                    # Manual: screenshot("custom-screenshot")
+    checkout.test/
+      001-find-before-L12-checkout-button.png
+      ...
+```
+
+### Automatic Screenshot Naming
+
+When `autoScreenshots` is enabled, filenames follow this format:
+
+`<seq>-<action>-<phase>-L<line>-<description>.png`
+
+| Component | Description | Example |
+|-----------|-------------|---------|
+| `seq` | Sequential number (001, 002, ...) | `001` |
+| `action` | Command name | `click`, `type`, `find` |
+| `phase` | Before, after, or error | `before`, `after` |
+| `L<line>` | Line number from test file | `L42` |
+| `description` | Element description or action target | `submit-button` |
+
+<Note>
+  The screenshot folder for each test file is automatically cleared when the test starts. This ensures you only see screenshots from the most recent test run.
+</Note>
+
+## Examples
+
+### Basic Screenshot
+
+```javascript
+// Capture a screenshot with auto-generated filename
+const screenshotPath = await testdriver.screenshot();
+console.log('Screenshot saved to:', screenshotPath);
+```
+
+### Custom Filename
+
+```javascript
+// Save with a descriptive filename
+await testdriver.screenshot("login-page");
+// Saves to: .testdriver/screenshots/<test>/login-page.png
+
+await testdriver.screenshot("after-click");
+// Saves to: .testdriver/screenshots/<test>/after-click.png
+```
+
+### Debugging with Screenshots
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+
+describe("Login Flow", () => {
+  it("should log in successfully", async (context) => {
+    const testdriver = TestDriver(context);
+    
+    await testdriver.provision.chrome({
+      url: 'https://myapp.com/login',
+    });
+
+    // Capture initial state
+    await testdriver.screenshot();
+
+    // Fill in login form
+    const emailInput = await testdriver.find("email input");
+    await emailInput.click();
+    await testdriver.type("user@example.com");
+
+    // Capture state after typing
+    await testdriver.screenshot();
+
+    const passwordInput = await testdriver.find("password input");
+    await passwordInput.click();
+    await testdriver.type("password123");
+
+    // Capture before clicking login
+    await testdriver.screenshot();
+
+    const loginButton = await testdriver.find("Login button");
+    await loginButton.click();
+
+    // Capture after login attempt
+    await testdriver.screenshot();
+
+    const result = await testdriver.assert("dashboard is visible");
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+## Automatic Screenshots
+
+By default, TestDriver captures screenshots **automatically** before and after every command. This creates a complete visual timeline of your test execution without any additional code.
+
+### Enabling/Disabling
+
+```javascript
+// Auto-screenshots enabled by default
+const testdriver = TestDriver(context);
+
+// Explicitly disable if needed (not recommended)
+const testdriver = TestDriver(context, {
+  autoScreenshots: false
+});
+```
+
+### What Gets Captured
+
+Automatic screenshots are taken around these commands:
+- `find()` / `findAll()`
+- `click()` / `hover()` / `doubleClick()` / `rightClick()`
+- `type()` / `pressKeys()`
+- `scroll()`
+- `waitForText()` / `waitForImage()`
+- `focusApplication()`
+- `assert()` / `extract()` / `exec()`
+
+### Example Output
+
+For this test code:
+
+```javascript
+// Line 15: Find email input
+const emailInput = await testdriver.find("email input");
+// Line 16: Click it
+await emailInput.click();
+// Line 17: Type email
+await testdriver.type("user@example.com");
+```
+
+TestDriver automatically saves:
+
+```
+001-find-before-L15-email-input.png
+002-find-after-L15-email-input.png
+003-click-before-L16-email-input.png
+004-click-after-L16-email-input.png
+005-type-before-L17-userexamplecom.png
+006-type-after-L17-userexamplecom.png
+```
+
+If an error occurs, the phase will be `error` instead of `after`.
+
+## Best Practices
+
+<AccordionGroup>
+  <Accordion title="Let automatic screenshots do the work">
+    With `autoScreenshots: true`, you get comprehensive coverage without adding manual `screenshot()` calls. Only add manual screenshots for specific named checkpoints.
+  </Accordion>
+  
+  <Accordion title="Use screenshots for debugging flaky tests">
+    When a test fails intermittently, add screenshots at key steps to capture the actual screen state. This helps identify timing issues or unexpected UI states.
+  </Accordion>
+
+  <Accordion title="Capture before assertions">
+    Take a screenshot before making assertions. If the assertion fails, you'll have a visual record of what the screen looked like.
+    
+    ```javascript
+    await testdriver.screenshot();
+    const result = await testdriver.assert("checkout button is visible");
+    ```
+  </Accordion>
+
+  <Accordion title="Add to .gitignore">
+    Add `.testdriver/screenshots/` to your `.gitignore` to avoid committing screenshots to version control:
+    
+    ```
+    # .gitignore
+    .testdriver/screenshots/
+    ```
+  </Accordion>
+</AccordionGroup>
+
+## Viewing Saved Screenshots
+
+After saving screenshots during test execution, you can view them using TestDriver MCP commands. This is especially useful for debugging failed tests or verifying test behavior.
+
+### MCP Commands for Screenshot Viewing
+
+**List all saved screenshots:**
+
+```
+list_local_screenshots()
+```
+
+**View a specific screenshot:**
+
+```
+view_local_screenshot({ path: "/full/path/to/screenshot.png" })
+```
+
+These commands allow you to:
+- View screenshots from failed tests to understand what went wrong
+- Review test execution flow by examining screenshots in chronological order
+- Compare screenshots across test runs to identify flaky behavior
+
+<Note>
+  For detailed workflows and examples of using these MCP commands for debugging, see the [Debugging with Screenshots](/v7/debugging-with-screenshots) guide.
+</Note>
+
+## Related
+
+- [Debugging with Screenshots](/v7/debugging-with-screenshots) - View and analyze saved screenshots using MCP
+- [assert()](/v7/assert) - Make AI-powered assertions
+- [find()](/v7/find) - Locate elements on screen

package/ai/skills/testdriver:screenshots/SKILL.md

@@ -0,0 +1,184 @@
+---
+name: testdriver:screenshots
+description: Capture and manage screenshots during test execution
+---
+<!-- Generated from screenshots.mdx. DO NOT EDIT. -->
+
+## Overview
+
+TestDriver can capture screenshots manually at any point during a test, or automatically before and after every command. Screenshots are saved to a structured directory for easy debugging.
+
+## Manual Screenshots
+
+Use `testdriver.screenshot()` to capture the current screen:
+
+```javascript
+const path = await testdriver.screenshot();
+console.log('Saved to:', path);
+// .testdriver/screenshots/my-test/screenshot-1719849312345.png
+```
+
+### Options
+
+```javascript
+await testdriver.screenshot(filename?)
+```
+
+<ParamField path="filename" type="string">
+  Custom filename for the screenshot. `.png` is appended automatically if missing. If omitted, defaults to `screenshot-<timestamp>.png`.
+</ParamField>
+
+**Returns:** `Promise<string>` — the absolute file path of the saved screenshot.
+
+```javascript
+// Default filename
+await testdriver.screenshot();
+// → .testdriver/screenshots/my-test/screenshot-1719849312345.png
+
+// Custom filename
+await testdriver.screenshot('login-page');
+// → .testdriver/screenshots/my-test/login-page.png
+
+// With .png extension
+await testdriver.screenshot('dashboard-loaded.png');
+// → .testdriver/screenshots/my-test/dashboard-loaded.png
+```
+
+## Auto Screenshots
+
+Enable automatic screenshots before and after every command:
+
+```javascript
+const testdriver = new TestDriver({
+  autoScreenshots: true,
+});
+```
+
+<ParamField path="autoScreenshots" type="boolean" default={false}>
+  When `true`, captures a screenshot before and after every SDK command (`click`, `type`, `find`, `scroll`, `hover`, `pressKeys`, `assert`, `exec`, etc.). On error, an error-phase screenshot replaces the after-phase screenshot.
+</ParamField>
+
+### Filename Format
+
+Auto-screenshots follow this naming convention:
+
+```
+<seq>-<action>-<phase>-L<line>-<description>.png
+```
+
+| Part | Description | Example |
+|---|---|---|
+| `seq` | 3-digit zero-padded sequence number | `001` |
+| `action` | Command name | `click`, `type`, `find` |
+| `phase` | `before`, `after`, or `error` | `before` |
+| `L<line>` | Source line number from your test file | `L42` |
+| `description` | Sanitized from command arguments (max 30 chars) | `submit-button` |
+
+**Examples:**
+```
+001-find-before-L15-login-button.png
+002-find-after-L15-login-button.png
+003-click-before-L16-login-button.png
+004-click-after-L16-login-button.png
+005-type-before-L18-username-field.png
+006-type-error-L18-username-field.png
+```
+
+### Phases
+
+| Phase | When | Description |
+|---|---|---|
+| `before` | Before command executes | Captures the screen state before the action |
+| `after` | After successful command | Captures the result of the action |
+| `error` | After failed command | Captures the screen at the point of failure (replaces `after`) |
+
+## Screenshot Directory
+
+Screenshots are saved to:
+
+```
+<cwd>/.testdriver/screenshots/<testFileName>/
+```
+
+Where `<testFileName>` is the test file name without its extension. For example, a test at `tests/login.test.mjs` saves screenshots to `.testdriver/screenshots/login.test/`.
+
+### Directory Cleanup
+
+The screenshot directory for each test file is **automatically cleaned** at the start of a test run. This happens once per process per test file to prevent concurrent tests from the same file from interfering with each other.
+
+## Debug Screenshots
+
+Elements have a `saveDebugScreenshot()` method for debugging element detection:
+
+```javascript
+const el = await testdriver.find('submit button');
+
+// Save the screenshot that was used to detect this element
+const debugPath = await el.saveDebugScreenshot();
+console.log('Debug screenshot:', debugPath);
+// → ./debug-screenshot-1719849312345.png
+
+// Custom path
+await el.saveDebugScreenshot('./my-debug.png');
+```
+
+This saves the screenshot that was captured during the `find()` call, which can be useful for understanding what the AI "saw" when locating the element.
+
+## Complete Example
+
+```javascript
+import { describe, it, beforeAll, afterAll } from 'vitest';
+import TestDriver from 'testdriverai';
+
+describe('Screenshot Example', () => {
+  let testdriver;
+
+  beforeAll(async () => {
+    testdriver = new TestDriver({
+      autoScreenshots: true,   // capture every step
+    });
+    await testdriver.ready();
+    await testdriver.provision.chrome({ url: 'https://example.com' });
+  });
+
+  afterAll(async () => {
+    await testdriver.disconnect();
+  });
+
+  it('captures the login flow', async () => {
+    // Auto-screenshots capture before/after each command
+
+    // Manual screenshot for a specific moment
+    await testdriver.screenshot('initial-page-load');
+
+    const username = await testdriver.find('username input');
+    await username.click();
+    await testdriver.type('testuser@example.com');
+
+    await testdriver.screenshot('after-username-entry');
+
+    const password = await testdriver.find('password input');
+    await password.click();
+    await testdriver.type('password123');
+
+    await testdriver.find('login button').click();
+
+    await testdriver.screenshot('after-login-click');
+  });
+});
+```
+
+After running, your screenshot directory will contain:
+```
+.testdriver/screenshots/login-flow.test/
+├── initial-page-load.png
+├── 001-find-before-L18-username-input.png
+├── 002-find-after-L18-username-input.png
+├── 003-click-before-L19-username-input.png
+├── 004-click-after-L19-username-input.png
+├── 005-type-before-L20-testuser-example-com.png
+├── 006-type-after-L20-testuser-example-com.png
+├── after-username-entry.png
+├── 007-find-before-L24-password-input.png
+├── ...
+```