testdriverai 6.2.0 → 6.2.1

package/docs/apps/tauri-apps.mdx

@@ -0,0 +1,361 @@
+---
+title: "Tauri Apps"
+sidebarTitle: "Tauri Apps"
+description: "Testing Tauri Apps with TestDriver"
+icon: "https://tauri.app/favicon.svg"
+---
+
+> [Tauri](https://tauri.app/) is a framework for building tiny, fast binaries for all major desktop and mobile platforms. Developers can integrate any frontend framework that compiles to HTML, JavaScript, and CSS for building their user experience while leveraging languages such as Rust, Swift, and Kotlin for backend logic when needed.
+>
+> \
+> – [https://tauri.app/start](https://tauri.app/start/)
+
+## Testing Tauri apps with TestDriver
+
+In this guide, we'll leverage [Playwright](https://playwright.dev/) and the [TestDriver Playwright SDK](/getting-started/playwright) to convert the [Tauri Quick Start](https://tauri.app/start/create-project/) to TestDriver's selectorless, Vision AI.
+
+<Info>View Source: https://github.com/testdriverai/demo-tauri-app</Info>
+
+### Requirements
+
+To start testing your Tauri app with TestDriver, you need the following:
+
+<AccordionGroup>
+  <Accordion title="Create a TestDriver account">
+    <Steps>
+      <Step title="Create a TestDriver Account">
+        You will need a [Free TestDriver Account](https://app.testdriver.ai/team) to get an API key.
+
+        <Card title="Sign Up for TestDriver" icon="user-plus" horizontal href="https://app.testdriver.ai/team">
+
+        </Card>
+      </Step>
+      <Step title="Set up your environment">
+        Copy your API key from [your TestDriver dashboard](https://app.testdriver.ai/team), and set it as an environment variable.
+
+        <Tabs>
+          <Tab title="macOS / Linux">
+            Export an environment variable on macOS or Linux systems:
+
+            ```bash
+            export TD_API_KEY="your_api_key_here"
+            ```
+          </Tab>
+          <Tab title="Windows">
+            Export an environment variable in PowerShell:
+
+            ```powershell
+            setx TD_API_KEY "your_api_key_here"
+            ```
+          </Tab>
+        </Tabs>
+      </Step>
+    </Steps>
+
+  </Accordion>
+  <Accordion title="Create a Tauri project">
+    <Note>
+      Follow Tauri's [Create a Project](https://tauri.app/start/create-project/)
+      guide.
+    </Note>
+  </Accordion>
+  <Accordion title="Create a Playwright project">
+    <Info>
+      This is a condensed version of [Playwright's Installation Instructions](https://playwright.dev/docs/intro).
+
+      **If you're new to Playwright, you should follow their guide first.**
+    </Info>
+    In your Tauri project, run:
+
+    <Tabs>
+      <Tab title="npm">
+        ```bash
+        npm init playwright@latest
+        ```
+      </Tab>
+      <Tab title="yarn">
+        ```bash
+        yarn create playwright
+        ```
+      </Tab>
+      <Tab title="pnpm">
+        ```bash
+        pnpm create playwright
+        ```
+      </Tab>
+    </Tabs>
+    Select the following options when prompted:
+
+    ```console
+    ✔ Do you want to use TypeScript or JavaScript?
+    > TypeScript
+    ✔ Where to put your end-to-end tests?
+    > tests
+    ✔ Add a GitHub Actions workflow? (y/N)
+    > N
+    ✔ Install Playwright browsers (can be done manually via 'npx playwright install')? (Y/n)
+    > Y
+    ```
+
+  </Accordion>
+  <Accordion title="Install the TestDriver Playwright SDK">
+    `@testdriver.ai/playwright` is an AI-powered extension of `@playwright/test`.
+
+    <Tabs>
+      <Tab title="npm">
+        ```bash
+        npm install @testdriver.ai/playwright
+        ```
+      </Tab>
+      <Tab title="yarn">
+        ```bash
+        yarn add @testdriver.ai/playwright
+        ```
+      </Tab>
+      <Tab title="pnpm">
+        ```bash
+        pnpm add @testdriver.ai/playwright
+        ```
+      </Tab>
+    </Tabs>
+
+  </Accordion>
+</AccordionGroup>
+
+## Testing the Tauri Web App
+
+### Setup
+
+First, we need to modify the default Playwright configuration and our Tauri project to work together:
+
+<Steps>
+  <Step title="Configure Playwright to start the Tauri frontend">
+    In the `playwright.config.ts` file, we'll configure the [`webServer`](https://playwright.dev/docs/test-webserver)
+    to start the Tauri frontend for Playwright to test against:
+
+    ```typescript playwright.config.ts
+      /* Run your local dev server before starting the tests */
+      // [!code ++:8]
+      webServer: {
+        command: "npm run dev",
+        url: "http://localhost:1420",
+        reuseExistingServer: !process.env.CI,
+        env: {
+          VITE_PLAYWRIGHT: "true",
+        },
+      },
+    });
+    ```
+
+  </Step>
+  <Step title="Mock Tauri APIs">
+    Since we're testing the Tauri frontend, we need to [mock IPC Requests](https://tauri.app/develop/tests/mocking/)
+    to simulate `invoke` calls to the Rust backend:
+
+    ```html src/index.html
+    <body>
+      <div id="root"></div>
+      <!-- [!code ++:7] -->
+      <script type="module">
+        // This is set in playwright.config.ts to allow our tests to mock Tauri IPC `invoke` calls
+        if (import.meta.env.VITE_PLAYWRIGHT) {
+          const { mockIPC } = await import("@tauri-apps/api/mocks");
+          window.mockIPC = mockIPC;
+        }
+      </script>
+      <script type="module" src="/src/main.tsx"></script>
+    </body>
+    ```
+
+    We only need to do this once, as we'll be accessing `window.mockIPC` in our tests.
+
+  </Step>
+  <Step title="Create a new test file">
+    Create a new file (e.g. `tests/testdriver.spec.ts`) with:
+
+    ```typescript tests/testdriver.spec.ts
+    import type { mockIPC } from "@tauri-apps/api/mocks";
+    import { expect, test } from "@playwright/test";
+
+    test.beforeEach(async ({ page }) => {
+      await page.goto("http://localhost:1420");
+    });
+
+    test("should have title", async ({ page }) => {
+      await expect(page).toHaveTitle("Tauri + React + TypeScript");
+    });
+    ```
+
+  </Step>
+  <Step title="Run Playwright in UI Mode">
+    Now we're ready to run Playwright and start working on our tests:
+
+    <Tabs>
+      <Tab title="npm">
+        ```bash
+        npx playwright test --ui
+        ```
+      </Tab>
+      <Tab title="yarn">
+        ```bash
+        yarn playwright test --ui
+        ```
+      </Tab>
+      <Tab title="pnpm">
+        ```bash
+        pnpm exec playwright test --ui
+        ```
+      </Tab>
+    </Tabs>
+    ![Playwright UI Mode](https://playwright.dev/assets/ideal-img/ui-mode.4e54d6b.3598.png)
+
+    Click the <Icon icon="play" /> button to successfully run the tests in the UI.
+
+    <Tip>
+      Click the <Icon icon="eye" /> button to automatically re-run tests on save.
+    </Tip>
+
+  </Step>
+</Steps>
+
+### Usage
+
+#### Import the TestDriver Playwright SDK
+
+By changing **1 line**, we can add TestDriver's AI capabilities to Playwright:
+
+```typescript tests/testdriver.spec.ts
+import type { mockIPC } from "@tauri-apps/api/mocks";
+// [!code --]
+import { expect, test } from "@playwright/test";
+// [!code ++]
+import { expect, test } from "@testdriver.ai/playwright";
+
+// For type-safety of the mockIPC function
+declare global {
+  interface Window {
+    mockIPC: typeof mockIPC;
+  }
+}
+
+test.beforeEach(async ({ page }) => {
+  await page.goto("http://localhost:1420");
+});
+
+test("should have title", async ({ page }) => {
+  await expect(page).toHaveTitle("Tauri + React + TypeScript");
+});
+```
+
+Notice how we're able to continue using Playwright's API (`toHaveTitle`)
+with `@testdriver.ai/playwright`.
+
+The test continues to pass as before, so now we can update our test to use natural language instead of selectors.
+
+#### Assertions with `expect.toMatchPrompt`
+
+With Playwright, we would normally use a `getByRole` selector to assert the heading text:
+
+```typescript tests/example.spec.ts
+test("should have heading", async ({ page }) => {
+  await expect(
+    page.getByRole("heading", { name: "Installation" }),
+  ).toBeVisible();
+});
+```
+
+With TestDriver, we can use natural language with `toMatchPrompt` instead:
+
+```typescript tests/testdriver.spec.ts
+test("should have heading", async ({ page }) => {
+  // [!code --:3]
+  await expect(
+    page.getByRole("heading", { name: "Installation" }),
+  ).toBeVisible();
+  // [!code ++:1]
+  await expect(page).toMatchPrompt("Heading says 'Welcome to Tauri + React'");
+});
+```
+
+#### Agentic tests with `test.agent`
+
+With TestDriver, we can skip the test implementation **entirely** and let AI perform the test for us:
+
+<Steps>
+  <Step title="Mock the `greet` call">
+    First, we need to [`mock our invoke calls`](https://tauri.app/develop/tests/mocking/#ipc-requests),
+    since we're testing the frontend behavior and not our Tauri backend:
+
+    ```typescript tests/testdriver.spec.ts
+    test.beforeEach(async ({ page }) => {
+      await page.goto("http://localhost:1420");
+      // [!code ++:12]
+      await page.evaluate(() => {
+        // https://tauri.app/develop/tests/mocking/#mocking-commands-for-invoke
+        window.mockIPC((cmd, args) => {
+          switch (cmd) {
+            case "greet":
+              args = args as { name: string };
+              return `Hello, ${args.name}! You've been greeted from Rust!`;
+            default:
+              throw new Error(`Unsupported command: ${cmd}`);
+          }
+        });
+      });
+    });
+    ```
+
+  </Step>
+  <Step title="Add an Agentic Test">
+    Next, wrap a _prompt_ in `test.agent` to perform the test:
+
+    ```typescript tests/testdriver.spec.ts
+    test.describe("should greet with name", () => {
+      test.agent(`
+        - Enter the name "Tauri"
+        - Click the "Greet" button
+        - You should see the text "Hello, Tauri! You've been greeted from Rust!"
+      `);
+    });
+    ```
+
+  </Step>
+</Steps>
+
+#### Continued Reading
+
+[Learn more about TestDriver's Playwright SDK](/getting-started/playwright)
+
+## Testing the Tauri Desktop App
+
+We can use TestDriver and natural language to test our Tauri desktop app:
+
+<Steps>
+  <Step title="Run the Desktop App">
+    <Tabs>
+      <Tab title="npm">```bash npm run tauri dev ```</Tab>
+      <Tab title="yarn">```bash yarn tauri dev ```</Tab>
+      <Tab title="pnpm">```bash pnpm tauri dev ```</Tab>
+    </Tabs>
+  </Step>
+  <Step title="Continued Reading">
+    <Note>See [Desktop Apps](/apps/desktop-apps) for more information.</Note>
+  </Step>
+</Steps>
+
+## Testing the Tauri Mobile App
+
+We can use TestDriver and natural language to test our Tauri iOS app:
+
+<Steps>
+  <Step title="Run the Mobile App">
+    <Tabs>
+      <Tab title="npm">```bash npm run tauri ios dev ```</Tab>
+      <Tab title="yarn">```bash yarn tauri ios dev ```</Tab>
+      <Tab title="pnpm">```bash pnpm tauri ios dev ```</Tab>
+    </Tabs>
+  </Step>
+  <Step title="Continued Reading">
+    <Note>See [Mobile Apps](/apps/mobile-apps) for more information.</Note>
+  </Step>
+</Steps>

package/docs/cli/overview.mdx

@@ -15,11 +15,11 @@ npx testdriverai@latest <command> [options]
 
 ## Available commands
 
-|        Command         | Description                                                  |
-| :--------------------: | :----------------------------------------------------------- |
-| [`run`](/commands/run) | Executes a TestDriver test.                                  |
-| [`edit`](/commands/edit)         | Launch interactive mode.                                     |
-| [`help`](/commands/help)         | Displays help information for the CLI or a specific command. |
+|         Command          | Description                                                  |
+| :----------------------: | :----------------------------------------------------------- |
+|  [`run`](/commands/run)  | Executes a TestDriver test.                                  |
+| [`edit`](/commands/edit) | Launch interactive mode.                                     |
+| [`help`](/commands/help) | Displays help information for the CLI or a specific command. |
 
 ## Available Flags
 
@@ -27,9 +27,9 @@ npx testdriverai@latest <command> [options]
 | :------------------ | :----------------------------------------------------------------------------------------- |
 | `--heal`            | Launch exploratory mode and attemp to recover if an error or failing state is encountered. |
 | `--write`           | Ovewrite test file with new commands resulting from agentic testing                        |
-| `--headless`        | Run test without opening a browser window (useful for CI/CD environments)                  |
 | `--new`             | Create a new sandbox environment for the test run.                                         |
 | `--summary=<value>` | Output file where AI summary should be saved.                                              |
+| `--junit=<value>`   | Output file where junit report should be saved.                                            |
 
 ## Example usage
 

package/docs/commands/assert.mdx

@@ -22,6 +22,7 @@ The `assert` command validates that a specific condition is true. It ensures tha
 | -------- | --------- | ------------------------------------------------------------------------------------------------------------------ |
 | `expect` | `string`  | The condition to check. This should describe what you expect to see on the screen.                                 |
 | `async`  | `boolean` | (Optional) If set to `true`, the test will continue without waiting for the assertion to pass. Default is `false`. |
+| `invert` | `boolean` | (Optional) If set to `true`, will fail if the assertion passes.                                                    |
 
 ## Example usage
 

package/docs/commands/hover-text.mdx

@@ -24,6 +24,7 @@ The `hover-text` command is used to locate text on the screen based on a descrip
 | `description` | `string` | A description of the text and what it represents. The actual text itself shouldn't be included here.                                                                                                                        |
 |   `action`    | `string` | The action to take when the text is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. Also supports `drag-start` and `drag-end` for [dragging text](/commands/hover-image#drag-functionality). |
 |   `method`    |  `enum`  | The matching algorithm to use. Possible values are `turbo` (default) and `ai`.                                                                                                                                              |
+|   `timeout`   | `number` | **(Optional)** The duration in milliseconds to wait for the text to appear. Default is `5000` (5 seconds).                                                                                                                  |
 
 ## Example usage
 
@@ -32,11 +33,12 @@ command: hover-text
 text: Sign Up
 description: link in the header
 action: click
+timeout: 10000 # 10 seconds
 ```
 
 ## Gotchas
 
-- If the text can't be located, the command will fail. Ensure the text is visible on the screen and matches exactly.
+- If the text can't be located, it will internally call the [wait-for-text](/commands/wait-for-text) command and wait for the text to appear. The wait `timeout` is 5 seconds by default, and the command fails if the text is not found.
 - Variations in font size, style, or screen resolution may affect detection accuracy.
 
 ## Notes

package/docs/commands/match-image.mdx

@@ -18,10 +18,11 @@ The `match-image` command is used to locate an image on the screen by matching i
 
 ## Arguments
 
-| Argument | Type     | Description                                                                                                                                                                                                                   |
-| -------- | -------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `path`   | `string` | The path to the image that needs to be matched. The path needs to be relative to the current test file                                                                                                                        |
-| `action` | `string` | The action to take when the image is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. Also supports `drag-start` and `drag-end` for [dragging images](/commands/hover-image#drag-functionality) |
+| Argument | Type      | Description                                                                                                                                                                                                                   |
+| -------- | --------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `path`   | `string`  | The path to the image that needs to be matched. The path needs to be relative to the current test file                                                                                                                        |
+| `action` | `string`  | The action to take when the image is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. Also supports `drag-start` and `drag-end` for [dragging images](/commands/hover-image#drag-functionality) |
+| `invert` | `boolean` | (Optional) If set to `true`, the command will pass when the specified image is NOT detected. Default is `false`.                                                                                                              |
 
 ## Example usage
 

package/docs/commands/press-keys.mdx

@@ -28,8 +28,7 @@ The keys supported are the standard [Windows keys](https://learn.microsoft.com/e
 
 <Tabs>
   <Tab title="Modifiers">
-    - `ctrl`, `shift`, `alt`, `command`, `option`, `win` 
-    - Left/right variants:
+    - `ctrl`, `shift`, `alt`, `command`, `option`, `win` - Left/right variants:
     `ctrlleft`, `ctrlright`, `shiftleft`, `shiftright`, `altleft`, `altright`,
     `optionleft`, `optionright`, `winleft`, `winright`
   </Tab>
@@ -37,9 +36,7 @@ The keys supported are the standard [Windows keys](https://learn.microsoft.com/e
     - All ASCII printable characters: letters `a`–`z`, digits `0`–`9`, common
     punctuation, and `space`.
   </Tab>
-  <Tab title="Function keys">
-  - `f1`–`f24`
-  </Tab>
+  <Tab title="Function keys">- `f1`–`f24`</Tab>
   <Tab title="Navigation">
     - `up`, `down`, `left`, `right`, `home`, `end`, `pageup`, `pagedown`,
     `pgup`, `pgdn`
@@ -57,9 +54,10 @@ The keys supported are the standard [Windows keys](https://learn.microsoft.com/e
     `separator`
   </Tab>
   <Tab title="Media & Browser">
-    - `playpause`, `stop`, `nexttrack`, `prevtrack`, `volumedown`, `volumeup`, `volumemute` 
-    - `browserback`, `browserforward`, `browserrefresh`, `browsersearch`, `browserstop`, `browserfavorites`, `browserhome` 
-    - `launchapp1`, `launchapp2`, `launchmail`, `launchmediaselect`
+    - `playpause`, `stop`, `nexttrack`, `prevtrack`, `volumedown`, `volumeup`,
+    `volumemute` - `browserback`, `browserforward`, `browserrefresh`,
+    `browsersearch`, `browserstop`, `browserfavorites`, `browserhome` -
+    `launchapp1`, `launchapp2`, `launchmail`, `launchmediaselect`
   </Tab>
   <Tab title="IME / International">
     - `hangul`, `hanguel`, `hanja`, `junja`, `kana`, `kanji`, `modechange`,

package/docs/commands/scroll-until-image.mdx

@@ -18,13 +18,14 @@ The `scroll-until-image` command is used to scroll the screen in a specified dir
 
 ## Arguments
 
-|   Argument    |   Type   | Description                                                                                                             |
-| :-----------: | :------: | :---------------------------------------------------------------------------------------------------------------------- |
-| `description` | `string` | A description of the image and what it represents.                                                                      |
-|  `direction`  | `string` | (Optional) The direction to scroll. Available directions are: `up`, `down`, `left`, `right`. Defaults to `down`.        |
-|  `distance`   | `number` | (Optional) The maximum number of pixels to scroll before giving up. Default is `10000`.                                 |
-|   `method`    | `string` | (Optional) The method to use to scroll the page. Available methods are: `mouse` and `keyboard`. Defaults to `keyboard`. |
-|    `path`     | `string` | (Optional) The relative path to the image file that needs to be matched on the screen.                                  |
+|   Argument    |   Type    | Description                                                                                                             |
+| :-----------: | :-------: | :---------------------------------------------------------------------------------------------------------------------- |
+| `description` | `string`  | A description of the image and what it represents.                                                                      |
+|  `direction`  | `string`  | (Optional) The direction to scroll. Available directions are: `up`, `down`, `left`, `right`. Defaults to `down`.        |
+|  `distance`   | `number`  | (Optional) The maximum number of pixels to scroll before giving up. Default is `10000`.                                 |
+|   `method`    | `string`  | (Optional) The method to use to scroll the page. Available methods are: `mouse` and `keyboard`. Defaults to `keyboard`. |
+|    `path`     | `string`  | (Optional) The relative path to the image file that needs to be matched on the screen.                                  |
+|   `invert`    | `boolean` | (Optional) If set to `true`, the command will scroll until the specified image is NOT detected. Default is `false`.     |
 
 <Note>
   Use either the `description` or `path` argument to match an image on the

package/docs/commands/scroll-until-text.mdx

@@ -18,12 +18,13 @@ The `scroll-until-text` command is used to scroll the screen in a specified dire
 
 ## Arguments
 
-|  Argument   |   Type   | Description                                                                                                             |
-| :---------: | :------: | :---------------------------------------------------------------------------------------------------------------------- |
-|   `text`    | `string` | The text to find on the screen. Longer and unique are better. Note this is **case sensitive**                           |
-| `direction` | `string` | (Optional) The direction to scroll. Available directions are: `up`, `down`, `left`, `right`. Defaults to `down`.        |
-|  `method`   | `string` | (Optional) The method to use to scroll the page. Available methods are: `mouse` and `keyboard`. Defaults to `keyboard`. |
-| `distance`  | `number` | (Optional) The maximum number of pixels to scroll before giving up. Default is `10000`.                                 |
+|  Argument   |   Type    | Description                                                                                                             |
+| :---------: | :-------: | :---------------------------------------------------------------------------------------------------------------------- |
+|   `text`    | `string`  | The text to find on the screen. Longer and unique are better. Note this is **case sensitive**                           |
+| `direction` | `string`  | (Optional) The direction to scroll. Available directions are: `up`, `down`, `left`, `right`. Defaults to `down`.        |
+|  `method`   | `string`  | (Optional) The method to use to scroll the page. Available methods are: `mouse` and `keyboard`. Defaults to `keyboard`. |
+| `distance`  | `number`  | (Optional) The maximum number of pixels to scroll before giving up. Default is `10000`.                                 |
+|  `invert`   | `boolean` | (Optional) If set to `true`, the command will scroll until the specified text is NOT detected. Default is `false`.      |
 
 <Note>
 If the method is `keyboard` it just searches for the string by doing `ctrl + f`. 

package/docs/commands/wait-for-image.mdx

@@ -18,10 +18,11 @@ The `wait-for-image` command waits until the specified image is detected on the
 
 ## Arguments
 
-|   Argument    |   Type   | Description                                                                                               |
-| :-----------: | :------: | :-------------------------------------------------------------------------------------------------------- |
-| `description` | `string` | A description of the image.                                                                               |
-|   `timeout`   | `number` | (Optional) The duration in milliseconds to wait for the image to appear. Default is `10000` (10 seconds). |
+|   Argument    |   Type    | Description                                                                                                       |
+| :-----------: | :-------: | :---------------------------------------------------------------------------------------------------------------- |
+| `description` | `string`  | A description of the image.                                                                                       |
+|   `timeout`   | `number`  | (Optional) The duration in milliseconds to wait for the image to appear. Default is `10000` (10 seconds).         |
+|   `invert`    | `boolean` | (Optional) If set to `true`, the command will wait until the specified image is NOT detected. Default is `false`. |
 
 ## Example usage
 

package/docs/commands/wait-for-text.mdx

@@ -18,11 +18,12 @@ The `wait-for-text` command waits until the specified text is detected on the sc
 
 ## Arguments
 
-| Argument  |   Type   | Description                                                                                            |
-| :-------: | :------: | :----------------------------------------------------------------------------------------------------- |
-|  `text`   | `string` | The text to find on the screen.                                                                        |
-| `timeout` | `number` | (Optional) The duration in milliseconds to wait for the text to appear. Default is `5000` (5 seconds). |
-| `method`  |  `enum`  | (Optional) The matching algorithm to use. Possible values are `ai` and `turbo`. Default is `turbo`     |
+| Argument  |   Type    | Description                                                                                                      |
+| :-------: | :-------: | :--------------------------------------------------------------------------------------------------------------- |
+|  `text`   | `string`  | The text to find on the screen.                                                                                  |
+| `timeout` | `number`  | (Optional) The duration in milliseconds to wait for the text to appear. Default is `5000` (5 seconds).           |
+| `method`  |  `enum`   | (Optional) The matching algorithm to use. Possible values are `ai` and `turbo`. Default is `turbo`               |
+| `invert`  | `boolean` | (Optional) If set to `true`, the command will wait until the specified text is NOT detected. Default is `false`. |
 
 ## Example usage