@testdriverai/agent 7.8.0-test.38

package/docs/v7/_drafts/commands/exec.mdx

@@ -0,0 +1,276 @@
+---
+title: "exec"
+sidebarTitle: "exec"
+description: "Execute custom shell or PowerShell scripts within your tests."
+icon: "code"
+mode: "wide"
+---
+
+import Replay from "/snippets/tests/exec-shell-replay.mdx";
+import Example from "/snippets/tests/exec-shell-yaml.mdx";
+
+<Replay />
+<Example />
+
+## Description
+
+The `exec` command allows you to execute shell commands (Linux) or PowerShell commands (Windows) within your TestDriver tests. This is useful for launching applications, file operations, or performing system commands during a test.
+
+## Arguments
+
+| Argument |   Type   | Description                                                                                                                                                                        |
+| :------: | :------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+|  `lang`  | `string` | The language of the script to execute. Supported values are `sh` (Shell/Linux) or `pwsh` (PowerShell/Windows).                                                                     |
+| `output` | `string` | The variable name to store the result of the script. This variable can be accessed as `${OUTPUT.<var>}` in future steps.                                                           |
+|  `code`  | `string` | The script to execute.                                                                                                                                                             |
+| `silent` | `string` | Defaults to `false`. The command will print the output of the script. This is useful for suppressing unnecessary or private output in the test logs and it's useful for debugging. |
+
+## Example usage
+
+This example demonstrates how to use the `exec` command to launch a calculator application.
+
+```yaml calculator.yaml
+version: 6.0.0
+steps:
+  - commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          start /B calc.exe
+          timeout /t 5
+      - command: wait-for-text
+        text: "calculator"
+        timeout: 30000
+```
+
+## Additional details
+
+- Supported `lang` values are `sh` or `pwsh`:
+  - `sh` code is executed in the shell on Linux sandboxes.
+  - `pwsh` code is executed in PowerShell on Windows sandboxes.
+    - **Note:** You can also use `pwsh` in [lifecycle](/guide/lifecycle) scripts to install npm packages if you need them.
+    - Otherwise, the `pwsh` code can be used within test steps to launch applications or perform simple commands (like writing text to a file on the machine to perform a simple file upload).
+- The `output` argument captures stdout from your script.
+
+## Protips
+
+- The `result` variable is already available in your script, overwrite it to store the output as shown in the examples.
+- Do any handling of arrays or nested objects within your `js` script:
+  - ✅ `result = users[1].profile.firstName`
+  - ✅ `result = data.length > 0 ? data[0].userEmail : 'no user found'` if no data is found the value of output will be `null`
+  - ✅ `result = someTestUserEmail`
+  - ✅ `result = someTextToAssert`
+  - ✅ `result = someDescriptionOfAnImageToScrollTo`
+- Don't try to pass any non-string values to `output`:
+  - ❌ `result = [...users, ...values]`
+  - ❌ `result = {name: "Dale Earnhardt", starts: 676, wins: 76}`
+  - ❌ `result = [{user1: ...}, {user2: ...}]`
+
+---
+
+## Ways to use `exec`
+
+Here is an example using both `pwsh` and `js` contexts within a `prerun.yaml` script which creates a temporary email account and automatically clicks links found in received emails.
+
+```yaml ./lifecycle/prerun.yaml [expandable]
+version: 6.0.0
+steps:
+  - commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          npm install @sendgrid/mail
+      - command: exec
+        lang: js
+        output: accountData
+        code: |
+          const Mailjs = require("@cemalgnlts/mailjs");
+          const mailjs = new Mailjs();
+          let account = await mailjs.createOneAccount()
+          console.log("Account created:", account);
+          result = JSON.stringify(account.data)
+      - command: exec
+        lang: js
+        output: emailAddress
+        code: |
+          const accountData = ${OUTPUT.accountData};
+          result = accountData.username
+  - prompt: Enter the generated email into the email field
+    commands:
+      - command: hover-text
+        text: standard_user
+        description: email input field label
+        action: click
+      - command: type
+        text: ${OUTPUT.emailAddress}
+  - prompt: Wait for an email, extract links, and open each link
+    commands:
+      - command: exec
+        lang: js
+        code: |
+          const Mailjs = require("@cemalgnlts/mailjs");
+          const { JSDOM } = require('jsdom'); // To parse HTML and extract links
+
+          const accountData = ${OUTPUT.accountData};
+
+          const getLatestEmailAndClickLinks = async () => {
+            try {
+              // Initialize the Mailjs client
+              const mailjs = new Mailjs();
+
+              // Login to your account
+              await mailjs.login(accountData.username, accountData.password);
+
+              // Fetch list of messages
+              const messages = await mailjs.getMessages();
+
+              if (messages.length === 0) {
+                console.log('No emails found.');
+                return;
+              }
+
+              // Assuming the latest email is the first one in the list
+              const latestMessage = messages[0];
+
+              // Fetch the full details of the latest email
+              const fullMessage = await mailjs.getMessage(latestMessage.id);
+
+              console.log('Latest Email Details:', fullMessage);
+
+              // Parse the HTML content to extract links
+              const dom = new JSDOM(fullMessage.html);
+              const links = Array.from(dom.window.document.querySelectorAll('a')).map(a => a.href);
+
+              console.log('Found Links:', links);
+
+              // Click (fetch) every link using native fetch
+              for (const link of links) {
+                try {
+                  const response = await fetch(link);
+                  console.log('Clicked ${link}: ${response.status}');
+                } catch (linkError) {
+                  console.error('Error clicking ${link}:', linkError);
+                }
+              }
+
+            } catch (error) {
+              console.error('Error fetching latest email or clicking links:', error);
+            }
+          };
+
+          getLatestEmailAndClickLinks();
+```
+
+### Using `exec` pwsh commands in a test file
+
+In a test file, you can use the `pwsh` context directly:
+
+```yaml calculator.yaml highlight={5-8} [expandable]
+version: 6.0.0
+steps:
+  - prompt: launch a calculator
+    commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          start /B calc.exe
+          timeout /t 5
+      - command: wait-for-text
+        text: "calculator"
+        timeout: 30000
+  - prompt: performing the operation 2 + 2 = on the calculator that is opened
+    commands:
+      - command: focus-application
+        name: galculator
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: plus button on the calculator
+        action: click
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: equals button on the calculator
+        action: click
+```
+
+### One more option
+
+You can also save reusable snippets (like launching the calculator) to be inserted into a script later with the [`run`](/commands/run) command. That version would look something like this:
+
+```yaml snippets/launch-calculator.yaml [expandable]
+version: 6.0.0
+steps:
+  - prompt: launch a calculator
+    commands:
+      - command: exec
+        lang: pwsh
+        code: |
+        start /B calc.exe
+          timeout /t 5
+      - command: wait-for-text
+        text: "calculator"
+        timeout: 30000
+```
+
+Then in the test:
+
+```yaml calculator.yaml highlight={5-6} [expandable]
+version: 6.0.0
+steps:
+  - prompt: launch a calculator
+    commands:
+      - command: run
+        file: snippets/launch-calculator.yaml
+  - prompt: performing the operation 2 + 2 = on the calculator that is opened
+    commands:
+      - command: focus-application
+        name: galculator
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: plus button on the calculator
+        action: click
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: equals button on the calculator
+        action: click
+```
+
+### Don't try to run `js` within a test field
+
+This example will fail at runtime, so don't try to execute `js` context directly in a test file. Remember - use this in `prerun` to setup your test!
+
+```yaml badtestfile.yaml [expandable]
+version: 6.0.0
+commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          npm install -g axios json2csv
+  - prompt: fetch user data from API
+    commands:
+      - command: exec
+        output: user
+        lang: js
+        code: |
+          const axios = require('axios');
+          const { Parser } = require('json2csv');
+          const fs = require('fs');
+
+          const response = await axios.get('https://jsonplaceholder.typicode.com/users');
+          const parser = new Parser();
+          const csv = parser.parse(response.data);
+          fs.writeFileSync('users.csv', csv);
+          const user = response.data[0].name;
+          result = user;
+          console.log('username', user);
+    ...
+```
+
+This example will produce errors in the TestDriver output or CLI since the runner won't have access to the Node.js VM context.

package/docs/v7/_drafts/commands/focus-application.mdx

@@ -0,0 +1,44 @@
+---
+title: "focus-application"
+sidebarTitle: "focus-application"
+description: "Bring a specific application window to the foreground."
+icon: "window-restore"
+mode: "wide"
+---
+
+## Description
+
+The `focus-application` command displays a specific application window to the foreground. This ensures that subsequent commands interact with the correct application during a test.
+
+## Arguments
+
+| Argument | Type     | Description                                                                             |
+| -------- | -------- | --------------------------------------------------------------------------------------- |
+| `name`   | `string` | The name of the application to focus. This should match the application's display name. |
+
+## Example usage
+
+```yaml
+command: focus-application
+name: Google Chrome
+```
+
+## Protips
+
+- Ensure the application name matches the exact name displayed in your operating system's task manager.
+- Use this command at the start of a test or before interacting with an application to avoid focus-related issues.
+
+<Note>
+If the specified application isn't running, the command will fail. Ensure the application is open before using this command.
+For example, to launch chrome try using the exec command:
+
+```yaml
+- command: exec
+  lang: pwsh
+  code: start chrome
+```
+
+</Note>
+## Notes
+
+- The `focus-application` command is useful for multi-application workflows where you need to switch between different apps during a test.

package/docs/v7/_drafts/commands/hover-image.mdx

@@ -0,0 +1,69 @@
+---
+title: "hover-image"
+sidebarTitle: "hover-image"
+description: "Locate an image on the screen and perform an action."
+icon: "image"
+mode: "wide"
+---
+
+import Replay from "/snippets/tests/hover-image-replay.mdx";
+import Example from "/snippets/tests/hover-image-yaml.mdx";
+
+<Replay />
+<Example />
+
+## Description
+
+The `hover-image` command is used to locate an image on the screen based on a description and perform an action on it. This can include hovering, clicking, double-clicking or dragging the image.
+
+## Arguments
+
+| Argument      | Type     | Description                                                                                                                                                                       |
+| ------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `description` | `string` | A description of the image and what it represents. Don't include the image itself here.                                                                                           |
+| `action`      | `string` | The action to take when the image is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. Also supports `drag-start` and `drag-end` for dragging images |
+
+The `hover-image` command is useful for interacting with visual elements that can't be identified using text-based selectors.
+
+---
+
+## Drag functionality
+
+- The `drag-start` and `drag-end` actions are used to drag an image to a specific location. `drag-start` could be the image (or inner text element, in which case using `hover-text` is preferred) to drag and `drag-end` is the location is supposed to be dragged to. This could be coupled with [`hover-text`](/commands/hover-text) or [`match-image`](/commands/match-image) or just `hover-image`.
+
+### Example: Drag and Drop to upload an image
+
+```yaml drag-and-drop.yaml
+version: 6.0.0
+steps:
+  - prompt: click on the image of a shopping cart
+    commands:
+      - command: hover-image
+        description: a pink tree image in the file explorer
+        action: drag-start
+      - command: hover-text
+        text: Upload
+        description: The drop zone of the application
+        action: drag-end
+```
+
+---
+
+## Example usage
+
+```yaml
+command: hover-image
+description: search icon in the webpage content
+action: click
+```
+
+## Protips
+
+- Use clear and concise descriptions for the image to improve detection accuracy.
+- Ensure the action specified matches the intended interaction with the image.
+- If the hover-image command fails to locate an image based on the provided description, consider using the match-image command instead. The match-image command is specifically designed for scenarios where precise image matching is required, such as when the visual element can't be reliably described with text. It works by directly comparing the image on the screen with a reference image file.
+
+## Gotchas
+
+- If the image can't be located based on the description, the command will fail.
+- Ensure the screen resolution and scaling settings are consistent to avoid detection issues.

package/docs/v7/_drafts/commands/hover-text.mdx

@@ -0,0 +1,47 @@
+---
+title: "hover-text"
+sidebarTitle: "hover-text"
+description: "Hover or click on text elements based on a description."
+icon: "text"
+mode: "wide"
+---
+
+import Replay from "/snippets/tests/hover-text-replay.mdx";
+import Example from "/snippets/tests/hover-text-yaml.mdx";
+
+<Replay />
+<Example />
+
+## Description
+
+The `hover-text` command is used to locate text on the screen based on a description and perform an action on it. This can include hovering, clicking, right-clicking, or double-clicking the text.
+
+## Arguments
+
+|   Argument    |   Type   | Description                                                                                                                                                                                                                 |
+| :-----------: | :------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+|    `text`     | `string` | The text to find on the screen. Longer and unique is better.                                                                                                                                                                |
+| `description` | `string` | A description of the text and what it represents. The actual text itself shouldn't be included here.                                                                                                                        |
+|   `action`    | `string` | The action to take when the text is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. Also supports `drag-start` and `drag-end` for [dragging text](/commands/hover-image#drag-functionality). |
+|   `method`    |  `enum`  | The matching algorithm to use. Possible values are `turbo` (default) and `ai`.                                                                                                                                              |
+|   `timeout`   | `number` | **(Optional)** The duration in milliseconds to wait for the text to appear. Default is `5000` (5 seconds).                                                                                                                  |
+
+## Example usage
+
+```yaml
+command: hover-text
+text: Sign Up
+description: link in the header
+action: click
+timeout: 10000 # 10 seconds
+```
+
+## Gotchas
+
+- If the text can't be located, it will internally call the [wait-for-text](/commands/wait-for-text) command and wait for the text to appear. The wait `timeout` is 5 seconds by default, and the command fails if the text is not found.
+- Variations in font size, style, or screen resolution may affect detection accuracy.
+
+## Notes
+
+- The `hover-text` command is ideal for interacting with textual elements that can't be identified using other selectors.
+- Supported actions allow flexibility in how the text is interacted with during the test.

package/docs/v7/_drafts/commands/if.mdx

@@ -0,0 +1,53 @@
+---
+title: "if"
+sidebarTitle: "if"
+description: "Conditionally execute commands based on a specified condition."
+icon: "split"
+mode: "wide"
+---
+
+## Description
+
+The `if` command is used to conditionally execute a set of commands based on whether a specified condition evaluates to true or false. If the condition is true, the commands in the `then` block are executed. Otherwise, the commands in the `else` block are executed.
+
+## Arguments
+
+|  Argument   |        Type        | Description                                                                   |
+| :---------: | :----------------: | :---------------------------------------------------------------------------- |
+| `condition` |      `string`      | The condition to evaluate.                                                    |
+|   `then`    | `list of commands` | The commands to run if the condition is true.                                 |
+|   `else`    | `list of commands` | The commands to run if the condition is false. It is an **optional command**. |
+
+## Example usage
+
+```yaml
+- prompt: click on sign up, if cookie banner appears close it
+  commands:
+    - command: if
+      condition: the text "Accept Cookies" is visible
+      then:
+        - command: hover-text
+          text: Accept Cookies
+          description: cookie banner button
+          action: click
+      else:
+        - command: hover-text
+          text: Sign Up
+          description: link in the header
+          action: click
+```
+
+## Protips
+
+- Ensure the `condition` is clearly defined and evaluates correctly to avoid unexpected behavior.
+- Use descriptive `then` and `else` blocks to handle both outcomes effectively.
+
+## Gotchas
+
+- The else block is optional, if the `condition` fails and there is no `else` block, the execution will continue from the next available command.
+- If the `condition` is invalid or can't be evaluated, the command will fail.
+- Ensure all commands in the `then` and `else` blocks are valid and properly formatted.
+
+---
+
+The `if` command is useful for creating conditional logic in your tests, allowing for more dynamic and flexible workflows.

package/docs/v7/_drafts/commands/match-image.mdx

@@ -0,0 +1,67 @@
+---
+title: "match-image"
+sidebarTitle: "match-image"
+description: "Locate an image on the screen and perform an action."
+icon: "images"
+mode: "wide"
+---
+
+import Replay from "/snippets/tests/match-image-replay.mdx";
+import Example from "/snippets/tests/match-image-yaml.mdx";
+
+<Replay />
+<Example />
+
+## Description
+
+The `match-image` command is used to locate an image on the screen by matching it with a reference image file and performing an action (For example, click or hover) at its center. This command is particularly useful for interacting with elements that the AI has difficulty locating using descriptions or other methods.
+
+## Arguments
+
+| Argument | Type      | Description                                                                                                                                                                                                                   |
+| -------- | --------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `path`   | `string`  | The path to the image that needs to be matched. The path needs to be relative to the current test file                                                                                                                        |
+| `action` | `string`  | The action to take when the image is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. Also supports `drag-start` and `drag-end` for [dragging images](/commands/hover-image#drag-functionality) |
+| `invert` | `boolean` | (Optional) If set to `true`, the command will pass when the specified image is NOT detected. Default is `false`.                                                                                                              |
+
+## Example usage
+
+```yaml
+command: match-image
+path: screenshots/button.png
+action: click
+```
+
+## How it works
+
+- The `match-image` command takes a screenshot of the desktop and searches for the location of the reference image within the screenshot.
+- The matching logic looks for the most similar image within the screenshot, not an exact match. If the similarity is below ~80%, it will search additional scales. If no match is found, the command will fail.
+- Screenshots should be stored in the `testdriver/screenshots/` directory.
+
+## Protips
+
+- To create high-quality screenshots for matching:
+  - Download the video of the test and open it at **full or actual size** on your computer.
+  - Use a screenshot tool (like Cleanshot X) to capture the target element.
+  - Center the clickable element as much as possible within the screenshot.
+- Ensure the image file is clear and free of unnecessary visual noise to improve matching accuracy.
+
+## Gotchas
+
+- The screenshot image should be of the actual size of the image that's being match against.
+- If the image match is below ~80% similarity, the program tries to match will different scales, its recommended to have an actual size image.
+- Variations in screen resolution, scaling settings, or platform-specific UI differences may affect matching accuracy.
+- Ensure the image file is stored in the correct directory structure (`testdriver/screenshots/`) for dynamic resolution.
+- The `path`needs to be relative to the current test file's directory. For example, if the test file is at `testdriver/onboarding/login.yaml` and the image is present at the conventional `testdriver/screenshots/nepal-flag.png` it should be referrenced as
+
+```yaml highlight={2}
+command: match-image
+path: ../screenshots/nepal-flag.png
+action: click
+```
+
+## Notes
+
+- The `match-image` command is ideal for interacting with visual elements that can't be reliably described or located using other commands like `hover-image`.
+- This command supports flexible scaling to account for minor differences in image size or resolution.
+- Use this command as a fallback when other methods fail to locate the desired element.

package/docs/v7/_drafts/commands/press-keys.mdx

@@ -0,0 +1,87 @@
+---
+title: "press-keys"
+sidebarTitle: "press-keys"
+description: "Simulate typing a keyboard combination."
+icon: "keyboard"
+mode: "wide"
+---
+
+import Replay from "/snippets/tests/press-keys-replay.mdx";
+import Example from "/snippets/tests/press-keys-yaml.mdx";
+
+<Replay />
+<Example />
+
+## Description
+
+The `press-keys` command is used to simulate typing a keyboard combination. This is useful for triggering shortcuts or interacting with applications via keyboard input.
+
+## Arguments
+
+| Argument | Type                     | Description                       |
+| -------- | ------------------------ | :-------------------------------- |
+| `keys`   | YAML sequence of strings | A list of keys to press together. |
+
+---
+
+The keys supported are the standard [Windows keys](https://learn.microsoft.com/en-us/dotnet/api/system.windows.forms.keys?view=windowsdesktop-9.0). Supported values:
+
+<Tabs>
+  <Tab title="Modifiers">
+    - `ctrl`, `shift`, `alt`, `command`, `option`, `win` - Left/right variants:
+    `ctrlleft`, `ctrlright`, `shiftleft`, `shiftright`, `altleft`, `altright`,
+    `optionleft`, `optionright`, `winleft`, `winright`
+  </Tab>
+  <Tab title="Printable">
+    - All ASCII printable characters: letters `a`–`z`, digits `0`–`9`, common
+    punctuation, and `space`.
+  </Tab>
+  <Tab title="Function keys">- `f1`–`f24`</Tab>
+  <Tab title="Navigation">
+    - `up`, `down`, `left`, `right`, `home`, `end`, `pageup`, `pagedown`,
+    `pgup`, `pgdn`
+  </Tab>
+  <Tab title="Editing">
+    - `backspace`, `delete`, `del`, `insert`, `enter`, `return`, `tab`,
+    `escape`, `esc`
+  </Tab>
+  <Tab title="Locks & Print">
+    - `capslock`, `numlock`, `scrolllock`, `printscreen`, `prntscrn`, `prtsc`,
+    `prtscr`, `print`
+  </Tab>
+  <Tab title="Numpad">
+    - `num0`–`num9`, `add`, `subtract`, `multiply`, `divide`, `decimal`,
+    `separator`
+  </Tab>
+  <Tab title="Media & Browser">
+    - `playpause`, `stop`, `nexttrack`, `prevtrack`, `volumedown`, `volumeup`,
+    `volumemute` - `browserback`, `browserforward`, `browserrefresh`,
+    `browsersearch`, `browserstop`, `browserfavorites`, `browserhome` -
+    `launchapp1`, `launchapp2`, `launchmail`, `launchmediaselect`
+  </Tab>
+  <Tab title="IME / International">
+    - `hangul`, `hanguel`, `hanja`, `junja`, `kana`, `kanji`, `modechange`,
+    `nonconvert`, `convert`, `final`, `yen`
+  </Tab>
+  <Tab title="Other">- `help`, `select`, `sleep`, `space`</Tab>
+</Tabs>
+
+---
+
+## Example usage
+
+```yaml
+command: press-keys
+keys:
+  - ctrl
+  - space
+```
+
+## Protips
+
+- Use this command to trigger system-wide or application-specific shortcuts, such as the Start menu or the browser elements.
+- Ensure the application or system is in focus when using this command to avoid unexpected behavior.
+
+## Notes
+
+- The `press-keys` command is ideal for automating workflows that rely on keyboard shortcuts.