testdriverai 6.2.2 → 7.1.0

package/docs/v7/playwright.mdx

@@ -0,0 +1,342 @@
+---
+title: "Get Started with TestDriver & Playwright"
+sidebarTitle: "Playwright"
+tag: "NEW"
+description: "In this guide, you'll setup your TestDriver account, create a new Playwright project, and leverage TestDriver's AI to convert tests to natural language."
+icon: "masks-theater"
+---
+
+## Overview
+
+`@testdriver.ai/playwright` is a backwards-compatible wrapper around `@playwright/test` that uses TestDriver's Vision AI to:
+
+- Make [natural language assertions](#assertions-with-expect-tomatchprompt)
+- [Replace brittle selectors](#locating-elements-with-testdriver-locate) with natural language
+- [Perform actions](#performing-actions-with-testdriver-act) with a prompt
+- Test with an [automated agent](#agentic-tests-with-test-agent)
+
+We'll be incrementally converting Playwright's example test from the sample code:
+
+```typescript tests/example.spec.ts icon=square-js
+// Before
+test("get started link", async ({ page }) => {
+  await page.goto("https://playwright.dev/");
+  await page.getByRole("link", { name: "Get started" }).click();
+  await expect(
+    page.getByRole("heading", { name: "Installation" }),
+  ).toBeVisible();
+});
+```
+
+To using natural language with TestDriver:
+
+```typescript tests/example.spec.ts icon=square-js
+// After
+import { test } from "@testdriver.ai/playwright";
+
+test.describe("get started link", () => {
+  test.beforeEach(async ({ page }) => page.goto("https://playwright.dev/"));
+  test.agent(`
+    - Click the 'Get started' link
+    - Verify the 'Installation' heading is visible
+  `);
+});
+```
+
+## Prerequisites
+
+<Steps>
+  <Step title="Create a TestDriver Account">
+    You will need a [Free TestDriver Account](https://app.testdriver.ai/team) to get an API key.
+
+    <Card title="Sign Up for TestDriver" icon="user-plus" horizontal href="https://app.testdriver.ai/team">
+
+    </Card>
+
+  </Step>
+  <Step title="Set up your environment">
+    Copy your API key from [the TestDriver dashboard](https://app.testdriver.ai/team), and set it as an environment variable.
+
+    <Tabs>
+      <Tab title="macOS / Linux">
+        ```bash Export an environment variable on macOS or Linux systems
+        export TD_API_KEY="your_api_key_here"
+        ```
+      </Tab>
+      <Tab title="Windows">
+        ```powershell Export an environment variable in PowerShell
+        setx TD_API_KEY "your_api_key_here"
+        ```
+      </Tab>
+    </Tabs>
+
+  </Step>
+</Steps>
+
+## Setup Playwright
+
+<Steps>
+  <Step title="Initialize Playwright">
+    <Info>
+      This is a condensed version of [Playwright's Installation Instructions](https://playwright.dev/docs/intro).
+
+      **If you're new to Playwright, you should follow their guide first.**
+    </Info>
+    In a new folder or an existing, run:
+
+    <Tabs>
+      <Tab title="npm">
+        ```bash
+        npm init playwright@latest
+        ```
+      </Tab>
+      <Tab title="yarn">
+        ```bash
+        yarn create playwright
+        ```
+      </Tab>
+      <Tab title="pnpm">
+        ```bash
+        pnpm create playwright
+        ```
+      </Tab>
+    </Tabs>
+    Select the following options when prompted:
+
+    ```console
+    ✔ Do you want to use TypeScript or JavaScript?
+    > TypeScript
+    ✔ Where to put your end-to-end tests?
+    > tests
+    ✔ Add a GitHub Actions workflow? (y/N)
+    > N
+    ✔ Install Playwright browsers (can be done manually via 'npx playwright install')? (Y/n)
+    > Y
+    ```
+
+    Then, confirm Playwright works by running:
+
+    <Tabs>
+      <Tab title="npm">
+        ```bash
+        npx playwright test
+        ```
+      </Tab>
+      <Tab title="yarn">
+        ```bash
+        yarn playwright test
+        ```
+      </Tab>
+      <Tab title="pnpm">
+        ```bash
+        pnpm exec playwright test
+        ```
+      </Tab>
+    </Tabs>
+
+  </Step>
+</Steps>
+
+## Setup `@testdriver.ai/playwright`
+
+<Steps>
+  <Step title="Install TestDriver">
+    `@testdriver.ai/playwright` as a backwards-compatible wrapper around `@playwright/test`:
+
+    <Tabs>
+      <Tab title="npm">
+        ```bash
+        npm install @testdriver.ai/playwright
+        ```
+      </Tab>
+      <Tab title="yarn">
+        ```bash
+        yarn add @testdriver.ai/playwright
+        ```
+      </Tab>
+      <Tab title="pnpm">
+        ```bash
+        pnpm add @testdriver.ai/playwright
+        ```
+      </Tab>
+    </Tabs>
+
+  </Step>
+  <Step title="Run Playwright">
+    Before we start using TestDriver in our tests, run Playwright in [UI Mode](https://playwright.dev/docs/test-ui-mode):
+
+    <Tabs>
+      <Tab title="npm">
+        ```bash
+        npx playwright test --ui
+        ```
+      </Tab>
+      <Tab title="yarn">
+        ```bash
+        yarn playwright test --ui
+        ```
+      </Tab>
+      <Tab title="pnpm">
+        ```bash
+        pnpm exec playwright test --ui
+        ```
+      </Tab>
+    </Tabs>
+    ![Playwright UI Mode](https://playwright.dev/assets/ideal-img/ui-mode.4e54d6b.3598.png)
+
+    Clicking the ▶️ button should successfully run the tests in the UI,
+    just as they did before with `playwright test` in the CLI.
+
+  </Step>
+  <Step title="Import TestDriver">
+    For the sake of simplicity, we'll be working with one test file for now.
+
+    Open `tests/example.spec.ts` in your editor & rename the `@playwright/test`
+    import to `@testdriver.ai/playwright`:
+
+    ```typescript tests/example.spec.ts icon=square-js
+    import { test, expect } from '@playwright/test'; // [!code --]
+    import { test, expect } from '@testdriver.ai/playwright'; // [!code ++]
+    ```
+
+    Click the <Icon icon="play" /> button to run the test and verify everything still works.
+
+    <Tip>
+      Click the <Icon icon="eye" /> button to automatically re-run tests on save.
+    </Tip>
+
+  </Step>
+</Steps>
+
+## Usage
+
+Because TestDriver uses AI vision instead of selectors, we can use natural language for
+[assertions](#assertions-with-expect-tomatchprompt),
+[locating](#locating-elements-with-testdriver-locate),
+performing [actions](#performing-actions-with-testdriver-act),
+or even having an [agent](#agentic-tests-with-test-agent) test for you!
+
+### Assertions with `expect.toMatchPrompt`
+
+Replace `toBeVisible` with `toMatchPrompt` to assert that the element is visible on the screen:
+
+```typescript tests/example.spec.ts icon=square-js
+// [!code --:2]
+// Expects page to have a heading with the name of Installation.
+await expect(page.getByRole("heading", { name: "Installation" })).toBeVisible();
+// [!code ++:2]
+await expect(page).toMatchPrompt("'Installation' heading is visible");
+```
+
+Before, the test needed code comments to describe what the assertion is _actually checking_.
+
+With `toMatchPrompt`, natural language acts as a description, selector, and assertion in one
+
+<Tip>
+  TestDriver can reduce the amount of complexity in your tests, but you can
+  still "opt-in" to Playwright assertions and selectors if you need to (e.g.
+  validating accessibility with `page.getByRole`).
+</Tip>
+
+### Locating elements with `testdriver.locate`
+
+TestDriver can replace `data-testid`s, `getByRole`, and CSS selectors with natural language.
+
+First, update your test to get access to the `testdriver` fixture:
+
+```typescript tests/example.spec.ts icon=square-js
+// [!code --]
+test('get started link', async ({ page, }) => {
+// [!code ++]
+test('get started link', async ({ page, testdriver }) => {
+```
+
+Then, replace `getByRole` with `testdriver.locate`:
+
+```typescript tests/example.spec.ts icon=square-js
+test('get started link', async ({ page, testdriver }) => {
+  await page.goto('https://playwright.dev/');
+
+  // [!code --:2]
+  // Click the get started link.
+  await page.getByRole("link", { name: "Get started" }).click();
+  // [!code ++:2]
+  const link = await testdriver(page).locate("Get started link");
+  await link.click();
+
+  await expect(page).toMatchPrompt("'Installation' heading is visible");
+```
+
+Now, our test uses natural language to both describe & locate the element.
+
+<Tip>
+  In the example above, you can still use Playwright to assert that the element is indeed a link for accessibility:
+
+```typescript tests/example.spec.ts icon=square-js
+const link = await testdriver(page).locate("Get started link");
+// [!code ++]
+expect(link).toHaveRole("link");
+await link.click();
+```
+
+This way you can write user-centric tests _and_ validate the implementation.
+
+</Tip>
+
+### Performing actions with `testdriver.act`
+
+We can combine `locate` and `click` from the previous example into one line with `testdriver.act`:
+
+```typescript tests/example.spec.ts icon=square-js
+test("get started link", async ({ page, testdriver }) => {
+  await page.goto("https://playwright.dev/");
+
+  // [!code --:2]
+  const link = await testdriver(page).locate("Get started link");
+  await link.click();
+  // [!code ++]
+  await testdriver(page).act("Click the 'Get started' link");
+
+  await expect(page).toMatchPrompt("'Installation' heading is visible");
+});
+```
+
+Now the test uses the page the way a user would!
+
+### Agentic tests with `test.agent`
+
+TestDriver can automatically perform the entire test for you with an AI agent:
+
+```typescript tests/example.spec.ts icon=square-js
+// [!code --:6]
+test("get started link", async ({ page, testdriver }) => {
+  await page.goto("https://playwright.dev/");
+  await testdriver(page).act("Click the 'Get started' link");
+  await expect(page).toMatchPrompt("'Installation' heading is visible");
+});
+
+// [!code ++:7]
+test.describe("get started link", () => {
+  test.beforeEach(async ({ page }) => page.goto("https://playwright.dev/"));
+  test.agent(`
+    - Click the 'Get started' link
+    - Verify the 'Installation' heading is visible
+  `);
+});
+```
+
+Instead of writing the test implementation, we've used [`test.describe`](https://playwright.dev/docs/api/class-test#test-describe) to describe the test still,
+but replaced the `test` itself with `test.agent`.
+
+<Tip>
+  Use `test.beforeEach` to prepare the page for the agent (e.g.
+  [`page.goto`](https://playwright.dev/docs/api/class-page#page-goto), calling
+  an API to create a user). Use
+  [`test.afterEach`](https://playwright.dev/docs/api/class-test#test-after-each)
+  to clean up after the agent (e.g. `page.close`) or perform additional logic
+  (e.g. clearing the session).
+</Tip>
+
+## Conclusion
+
+With `@testdriver.ai/playwright`, you can use as much or as little of Playwright's _or_ TestDriver's API as you need to validate correctness. It's up to you!

package/docs/v7/presets/chrome-extension.mdx

@@ -0,0 +1,223 @@
+---
+title: "Chrome Extension Testing"
+sidebarTitle: "Chrome Extensions"
+description: "Test Chrome extensions with Chrome for Testing"
+icon: "puzzle-piece"
+---
+
+## Overview
+
+Test Chrome extensions by loading them into Chrome for Testing. This preset uses the `launchChromeExtension()` helper to launch Chrome with a specific extension loaded by its Chrome Web Store ID.
+
+## Quick Start
+
+```javascript
+import { test } from 'vitest';
+import TestDriver from 'testdriverai';
+import { 
+  runPrerunChromeExtension, 
+  runPostrun 
+} from 'testdriverai/testdriver/acceptance-sdk/setup/lifecycleHelpers.mjs';
+
+test('test chrome extension', async () => {
+  const client = await TestDriver.create({
+    apiKey: process.env.TD_API_KEY,
+    os: "linux",
+    verbosity: 1,
+  });
+
+  // Launch Chrome with extension loaded
+  // Extension ID from Chrome Web Store
+  await runPrerunChromeExtension(client, "cjpalhdlnbpafiamejdnhcphjbkeiagm");
+
+  // Your test code here
+  await client.focusApplication("Google Chrome");
+  
+  // ... test extension functionality
+
+  await runPostrun(client);
+  await client.cleanup();
+});
+```
+
+## Finding Extension IDs
+
+Extension IDs can be found in the Chrome Web Store URL:
+
+```
+https://chrome.google.com/webstore/detail/ublock-origin/cjpalhdlnbpafiamejdnhcphjbkeiagm
+                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+                                                      This is the extension ID
+```
+
+## Popular Extensions
+
+| Extension | ID |
+|-----------|---|
+| uBlock Origin | `cjpalhdlnbpafiamejdnhcphjbkeiagm` |
+| React Developer Tools | `fmkadmapgofadopljbjfkapdkoienihi` |
+| Redux DevTools | `lmhkpmbekcpmknklioeibfkpmmfibljd` |
+| Bitwarden | `nngceckbapebfimnlniiiahkandclblb` |
+
+## Loading Multiple Extensions
+
+Load multiple extensions by separating IDs with commas:
+
+```javascript
+await launchChromeExtension(
+  client, 
+  "cjpalhdlnbpafiamejdnhcphjbkeiagm,nngceckbapebfimnlniiiahkandclblb"
+);
+```
+
+## Complete Example
+
+```javascript
+import { describe, it, beforeAll, afterAll } from 'vitest';
+import TestDriver from 'testdriverai';
+import { 
+  runPrerunChromeExtension, 
+  runPostrun 
+} from 'testdriverai/testdriver/acceptance-sdk/setup/lifecycleHelpers.mjs';
+
+describe('Chrome Extension Testing', () => {
+  let client;
+  let dashcamUrl;
+
+  beforeAll(async () => {
+    client = await TestDriver.create({
+      apiKey: process.env.TD_API_KEY,
+      os: "linux",
+      verbosity: 1,
+    });
+
+    // Load uBlock Origin extension
+    await runPrerunChromeExtension(client, "cjpalhdlnbpafiamejdnhcphjbkeiagm");
+  });
+
+  afterAll(async () => {
+    if (client) {
+      dashcamUrl = await runPostrun(client);
+      await client.cleanup();
+    }
+  });
+
+  it('should verify extension is loaded', async () => {
+    await client.focusApplication("Google Chrome");
+
+    // Navigate to a page
+    const element = await client.find("TestDriver.ai Sandbox");
+    expect(element.found()).toBe(true);
+
+    // Test extension-specific functionality
+    // For example, checking if ads are blocked with uBlock
+  });
+
+  it('should access extension popup', async () => {
+    await client.focusApplication("Google Chrome");
+    
+    // Open extension management
+    await client.exec(
+      "sh",
+      `xdotool key --clearmodifiers ctrl+shift+e`,
+      5000,
+      true
+    );
+
+    // Wait for extensions page
+    await new Promise((resolve) => setTimeout(resolve, 2000));
+  });
+});
+```
+
+## Environment Setup
+
+Chrome for Testing is pre-installed in the E2B sandbox environment at:
+
+```
+/opt/chrome-for-testing/chrome
+/usr/local/bin/chrome-for-testing (symlink)
+```
+
+## Direct API Usage
+
+Use the lower-level API for more control:
+
+```javascript
+import { launchChromeExtension } from 'testdriverai/testdriver/acceptance-sdk/setup/lifecycleHelpers.mjs';
+
+// Launch with specific extension and URL
+await launchChromeExtension(
+  client,
+  "cjpalhdlnbpafiamejdnhcphjbkeiagm",
+  "https://example.com"
+);
+```
+
+## Testing Extension Features
+
+### Test Extension Popup
+
+```javascript
+it('opens extension popup', async () => {
+  // Click extension icon (varies by extension)
+  await client.click('extension icon in toolbar');
+  
+  // Interact with popup
+  const popup = await client.find('extension popup window');
+  expect(popup.found()).toBe(true);
+});
+```
+
+### Test Extension Settings
+
+```javascript
+it('configures extension settings', async () => {
+  // Right-click extension icon
+  await client.rightClick('extension icon');
+  
+  // Click options
+  await client.click('Options');
+  
+  // Configure settings
+  await client.click('Enable feature X');
+  await client.click('Save');
+});
+```
+
+### Test Content Scripts
+
+```javascript
+it('verifies content script injection', async () => {
+  // Navigate to a page
+  await client.exec('sh', 'xdotool key ctrl+l', 5000, true);
+  await client.type('https://example.com');
+  await client.pressKeys('Enter');
+  
+  // Check for extension-injected elements
+  const injected = await client.find('element added by extension');
+  expect(injected.found()).toBe(true);
+});
+```
+
+## Troubleshooting
+
+### Extension Not Loading
+
+1. Verify the extension ID is correct
+2. Check Chrome for Testing is installed in the sandbox
+3. Ensure extension is compatible with Chrome for Testing version
+
+### Extension Permissions
+
+Some extensions may require additional permissions or setup. You may need to:
+
+1. Navigate to `chrome://extensions/`
+2. Enable developer mode
+3. Grant required permissions
+
+## See Also
+
+- [Web Apps (Chrome)](/v7/presets/chrome) - Regular Chrome browser testing
+- [Desktop Apps (Electron)](/v7/presets/electron) - Electron app testing
+- [Lifecycle Helpers](/v7/guides/lifecycle) - Prerun/postrun functions