@testdriverai/agent 7.8.0-canary.10

package/docs/v6/bugs/jira.mdx

@@ -0,0 +1,232 @@
+---
+title: "Reporting Failed Tests to Jira"
+sidebarTitle: "Jira"
+description: "Step-by-step instructions to integrate TestDriver with Jira for automated test failure tracking in CI/CD workflows."
+icon: "jira"
+---
+
+# Automating Jira ticket creation for test failures using TestDriver CLI and Jira GitHub Actions
+
+This guide explains how to integrate the **TestDriver CLI** with the **Jira GitHub Action** to automatically create a Jira ticket whenever a TestDriver test fails. This workflow ensures that test failures are tracked in Jira, enabling teams to address issues promptly.
+
+---
+
+## Workflow overview
+
+<Steps>
+  <Step title="Run Tests with TestDriver CLI">
+    Use the TestDriver CLI with the --headless flag to execute your test suite
+    in a headless environment.
+  </Step>
+  <Step title="Check for Test Failures">
+    Capture the test results and determine if any tests failed.
+  </Step>
+  <Step title="Create a Jira Ticket">
+    Use the Jira GitHub Action to create a new ticket for each test failure,
+    including relevant details such as the test name, failure reason, and logs.
+  </Step>
+</Steps>
+---
+
+## Prerequisites
+
+1. **TestDriver API Key**: Store your API key as a GitHub secret (for example, `TD_API_KEY`).
+2. **Jira API Token**: Generate an API token from your Jira account and store it as a GitHub secret (for example, `JIRA_API_TOKEN`).
+3. **Jira Base URL**: Your Jira instance URL (for example, `https://yourcompany.atlassian.net`).
+4. **Jira Project Key**: The key of the Jira project where tickets will be created (for example, `TEST`).
+5. **TestDriver CLI**: The TestDriver CLI must be installed and configured in your workflow.
+
+---
+
+## GitHub Actions workflow
+
+Here's a complete workflow that integrates TestDriver and Jira:
+
+### Workflow File:
+
+```yaml .github/workflows/testdriver-jira.yaml [expandable]
+name: TestDriver with Jira Integration
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  run-tests:
+    name: Run Tests with TestDriver
+    runs-on: windows-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "18"
+
+      - name: Run TestDriver
+        id: testdriver
+        shell: powershell
+        run: |
+          $exitCode = 0
+          try {
+            npx testdriverai@latest run testdriver/test.yaml --summary="testdriver-summary.txt"
+          } catch {
+            $exitCode = $LASTEXITCODE
+          }
+
+          # Set outputs for next steps
+          echo "exit_code=$exitCode" >> $env:GITHUB_OUTPUT
+          if ($exitCode -ne 0) {
+            echo "success=false" >> $env:GITHUB_OUTPUT
+          } else {
+            echo "success=true" >> $env:GITHUB_OUTPUT
+          }
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+
+      - name: Check for Test Failures
+        id: check-failures
+        shell: powershell
+        run: |
+          if ("${{ steps.testdriver.outputs.success }}" -eq "false") {
+            echo "TestDriver tests failed."
+            echo "failure=true" >> $env:GITHUB_ENV
+            
+            # Copy the output file for Jira ticket
+            Copy-Item testdriver-summary.txt failure-details.txt
+          } else {
+            echo "All tests passed."
+            echo "failure=false" >> $env:GITHUB_ENV
+          }
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: testdriver-results
+          path: |
+            testdriver-summary.txt
+            failure-details.txt
+
+  create-jira-ticket:
+    name: Create Jira Ticket for Test Failures
+    needs: run-tests
+    runs-on: ubuntu-latest
+    if: needs.run-tests.outputs.failure == 'true'
+    steps:
+      - name: Download test results
+        uses: actions/download-artifact@v4
+        with:
+          name: testdriver-results
+
+      - name: Create Jira Ticket
+        uses: atlassian/gajira-create@v3
+        with:
+          url: ${{ secrets.JIRA_BASE_URL }}
+          user: ${{ secrets.JIRA_USERNAME }}
+          api-token: ${{ secrets.JIRA_API_TOKEN }}
+          project: TEST # Replace with your Jira project key
+          summary: "TestDriver Test Failure in ${{ github.repository }}"
+          description: |
+            A test failure occurred during the TestDriver workflow.
+
+            **Repository**: ${{ github.repository }}
+            **Branch**: ${{ github.ref }}
+            **Commit**: ${{ github.sha }}
+            **Workflow Run**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+
+            **Failure Details**:
+            $(cat failure-details.txt)
+
+            Please investigate the issue and resolve it promptly.
+          issuetype: Bug
+```
+
+---
+
+## Workflow steps explained
+
+### 1. **Run Tests with TestDriver CLI**
+
+This step runs the TestDriver CLI with the `testdriver/test.yaml` file.
+
+```yaml testdriver/test.yaml
+- name: Run TestDriver
+  id: testdriver
+  run: npx testdriverai@latest run testdriver/test.yaml
+  env:
+    TD_API_KEY: ${{ secrets.TD_API_KEY }}
+```
+
+---
+
+### 2. **Check for test failures**
+
+This step checks the exit code and success status from the TestDriver CLI run. If the tests failed, it sets an environment variable (`failure=true`) to trigger the Jira ticket creation step and prepares the output file for the Jira ticket.
+
+```yaml testdriver/test.yaml
+- name: Check for Test Failures
+  id: check-failures
+  run: |
+    if ("${{ steps.testdriver.outputs.success }}" -eq "false") {
+      echo "TestDriver tests failed."
+      echo "failure=true" >> $env:GITHUB_ENV
+      
+      # Copy the output file for Jira ticket
+      Copy-Item testdriver-output.txt failure-details.txt
+    } else {
+      echo "All tests passed."
+      echo "failure=false" >> $env:GITHUB_ENV
+    }
+  shell: powershell
+```
+
+---
+
+### 3. **Create a Jira ticket**
+
+If any tests failed, the `create-jira-ticket` job uses the `atlassian/gajira-create` action to create a new Jira ticket. The ticket includes:
+
+- **Summary**: A brief description of the failure with repository information.
+- **Description**: Detailed information about the failure, including repository details, workflow run links, and the captured test output.
+
+```yaml testdriver/test.yaml
+- name: Create Jira Ticket
+  uses: atlassian/gajira-create@v3
+  with:
+    url: ${{ secrets.JIRA_BASE_URL }}
+    user: ${{ secrets.JIRA_USERNAME }}
+    api-token: ${{ secrets.JIRA_API_TOKEN }}
+    project: TEST # Replace with your Jira project key
+    summary: "TestDriver Test Failure in ${{ github.repository }}"
+    description: |
+      A test failure occurred during the TestDriver workflow.
+
+      **Repository**: ${{ github.repository }}
+      **Branch**: ${{ github.ref }}
+      **Commit**: ${{ github.sha }}
+      **Workflow Run**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+
+      **Failure Details**:
+      $(cat failure-details.txt)
+
+      Please investigate the issue and resolve it promptly.
+    issuetype: Bug
+```
+
+---
+
+## Secrets configuration
+
+Add the following secrets to your GitHub repository:
+
+1. **`TD_API_KEY`**: Your TestDriver API key.
+2. **`JIRA_API_TOKEN`**: Your Jira API token.
+3. **`JIRA_BASE_URL`**: Your Jira instance URL (for example, `https://yourcompany.atlassian.net`).
+4. **`JIRA_USERNAME`**: Your Jira account email.
+
+---

package/docs/v6/cli/overview.mdx

@@ -0,0 +1,66 @@
+---
+title: "TestDriver CLI Flags"
+sidebarTitle: "Flags"
+description: "Learn about the TestDriver CLI, its commands, and how to use it effectively."
+icon: "flag"
+---
+
+The `testdriverai` CLI is the primary interface for running, managing, and debugging TestDriver scripts. It provides commands to execute tests, validate configurations, and interact with the TestDriver framework.
+
+## Usage
+
+```bash
+npx testdriverai@latest <command> [options]
+```
+
+## Available commands
+
+|         Command          | Description                                                  |
+| :----------------------: | :----------------------------------------------------------- |
+|  [`run`](/commands/run)  | Executes a TestDriver test.                                  |
+| [`edit`](/commands/edit) | Launch interactive mode.                                     |
+| [`help`](/commands/help) | Displays help information for the CLI or a specific command. |
+
+## Available Flags
+
+| Flag                | Description                                                                                |
+| :------------------ | :----------------------------------------------------------------------------------------- |
+| `--heal`            | Launch exploratory mode and attemp to recover if an error or failing state is encountered. |
+| `--write`           | Ovewrite test file with new commands resulting from agentic testing                        |
+| `--new`             | Create a new sandbox environment for the test run.                                         |
+| `--summary=<value>` | Output file where AI summary should be saved.                                              |
+| `--junit=<value>`   | Output file where junit report should be saved.                                            |
+
+## Example usage
+
+### Running a test script
+
+```bash
+npx testdriverai@latest run path/to/testdriver.yaml
+```
+
+This command runs the specified TestDriver script, executing all steps defined in the file.
+
+### Initializing a new project
+
+```bash
+npx testdriverai@latest init
+```
+
+This command sets up a new TestDriver project in the current directory, creating the necessary folder structure and configuration files.
+
+### Getting help
+
+```bash
+npx testdriverai@latest help
+```
+
+This command displays general help information. To get help for a specific command, use:
+
+```bash
+npx testdriverai@latest help <command>
+```
+
+## Protips
+
+- Combine [`run`](/commands/run) and `--headless` with CI/CD pipelines to automate test execution.

package/docs/v6/commands/assert.mdx

@@ -0,0 +1,45 @@
+---
+title: "assert"
+sidebarTitle: "assert"
+description: "Validate conditions during a test using the assert command."
+icon: "check"
+mode: "wide"
+---
+
+import Replay from "/snippets/tests/assert-replay.mdx";
+import Example from "/snippets/tests/assert-yaml.mdx";
+
+<Replay />
+<Example />
+
+## Description
+
+The `assert` command validates that a specific condition is true. It ensures that a task completed successfully by checking the screen for the expected outcome. If the condition isn't met, the test will fail.
+
+## Arguments
+
+| Argument | Type      | Description                                                                                                        |
+| -------- | --------- | ------------------------------------------------------------------------------------------------------------------ |
+| `expect` | `string`  | The condition to check. This should describe what you expect to see on the screen.                                 |
+| `async`  | `boolean` | (Optional) If set to `true`, the test will continue without waiting for the assertion to pass. Default is `false`. |
+| `invert` | `boolean` | (Optional) If set to `true`, will fail if the assertion passes.                                                    |
+
+## Example usage
+
+```yaml
+command: assert
+expect: the video is playing
+```
+
+### Example with `async`
+
+```yaml
+command: assert
+expect: There is no error message
+async: true
+```
+
+## Notes
+
+- Use `async: true` to speed up tests by allowing non-blocking assertions. However, the test will still fail if the condition isn't met.
+- Ensure the `expect` string clearly describes the condition to avoid ambiguity.

package/docs/v6/commands/exec.mdx

@@ -0,0 +1,276 @@
+---
+title: "exec"
+sidebarTitle: "exec"
+description: "Execute custom shell or PowerShell scripts within your tests."
+icon: "code"
+mode: "wide"
+---
+
+import Replay from "/snippets/tests/exec-shell-replay.mdx";
+import Example from "/snippets/tests/exec-shell-yaml.mdx";
+
+<Replay />
+<Example />
+
+## Description
+
+The `exec` command allows you to execute shell commands (Linux) or PowerShell commands (Windows) within your TestDriver tests. This is useful for launching applications, file operations, or performing system commands during a test.
+
+## Arguments
+
+| Argument |   Type   | Description                                                                                                                                                                        |
+| :------: | :------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+|  `lang`  | `string` | The language of the script to execute. Supported values are `sh` (Shell/Linux) or `pwsh` (PowerShell/Windows).                                                                     |
+| `output` | `string` | The variable name to store the result of the script. This variable can be accessed as `${OUTPUT.<var>}` in future steps.                                                           |
+|  `code`  | `string` | The script to execute.                                                                                                                                                             |
+| `silent` | `string` | Defaults to `false`. The command will print the output of the script. This is useful for suppressing unnecessary or private output in the test logs and it's useful for debugging. |
+
+## Example usage
+
+This example demonstrates how to use the `exec` command to launch a calculator application.
+
+```yaml calculator.yaml
+version: 6.0.0
+steps:
+  - commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          start /B calc.exe
+          timeout /t 5
+      - command: wait-for-text
+        text: "calculator"
+        timeout: 30000
+```
+
+## Additional details
+
+- Supported `lang` values are `sh` or `pwsh`:
+  - `sh` code is executed in the shell on Linux sandboxes.
+  - `pwsh` code is executed in PowerShell on Windows sandboxes.
+    - **Note:** You can also use `pwsh` in [lifecycle](/guide/lifecycle) scripts to install npm packages if you need them.
+    - Otherwise, the `pwsh` code can be used within test steps to launch applications or perform simple commands (like writing text to a file on the machine to perform a simple file upload).
+- The `output` argument captures stdout from your script.
+
+## Protips
+
+- The `result` variable is already available in your script, overwrite it to store the output as shown in the examples.
+- Do any handling of arrays or nested objects within your `js` script:
+  - ✅ `result = users[1].profile.firstName`
+  - ✅ `result = data.length > 0 ? data[0].userEmail : 'no user found'` if no data is found the value of output will be `null`
+  - ✅ `result = someTestUserEmail`
+  - ✅ `result = someTextToAssert`
+  - ✅ `result = someDescriptionOfAnImageToScrollTo`
+- Don't try to pass any non-string values to `output`:
+  - ❌ `result = [...users, ...values]`
+  - ❌ `result = {name: "Dale Earnhardt", starts: 676, wins: 76}`
+  - ❌ `result = [{user1: ...}, {user2: ...}]`
+
+---
+
+## Ways to use `exec`
+
+Here is an example using both `pwsh` and `js` contexts within a `prerun.yaml` script which creates a temporary email account and automatically clicks links found in received emails.
+
+```yaml ./lifecycle/prerun.yaml [expandable]
+version: 6.0.0
+steps:
+  - commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          npm install @sendgrid/mail
+      - command: exec
+        lang: js
+        output: accountData
+        code: |
+          const Mailjs = require("@cemalgnlts/mailjs");
+          const mailjs = new Mailjs();
+          let account = await mailjs.createOneAccount()
+          console.log("Account created:", account);
+          result = JSON.stringify(account.data)
+      - command: exec
+        lang: js
+        output: emailAddress
+        code: |
+          const accountData = ${OUTPUT.accountData};
+          result = accountData.username
+  - prompt: Enter the generated email into the email field
+    commands:
+      - command: hover-text
+        text: standard_user
+        description: email input field label
+        action: click
+      - command: type
+        text: ${OUTPUT.emailAddress}
+  - prompt: Wait for an email, extract links, and open each link
+    commands:
+      - command: exec
+        lang: js
+        code: |
+          const Mailjs = require("@cemalgnlts/mailjs");
+          const { JSDOM } = require('jsdom'); // To parse HTML and extract links
+
+          const accountData = ${OUTPUT.accountData};
+
+          const getLatestEmailAndClickLinks = async () => {
+            try {
+              // Initialize the Mailjs client
+              const mailjs = new Mailjs();
+
+              // Login to your account
+              await mailjs.login(accountData.username, accountData.password);
+
+              // Fetch list of messages
+              const messages = await mailjs.getMessages();
+
+              if (messages.length === 0) {
+                console.log('No emails found.');
+                return;
+              }
+
+              // Assuming the latest email is the first one in the list
+              const latestMessage = messages[0];
+
+              // Fetch the full details of the latest email
+              const fullMessage = await mailjs.getMessage(latestMessage.id);
+
+              console.log('Latest Email Details:', fullMessage);
+
+              // Parse the HTML content to extract links
+              const dom = new JSDOM(fullMessage.html);
+              const links = Array.from(dom.window.document.querySelectorAll('a')).map(a => a.href);
+
+              console.log('Found Links:', links);
+
+              // Click (fetch) every link using native fetch
+              for (const link of links) {
+                try {
+                  const response = await fetch(link);
+                  console.log('Clicked ${link}: ${response.status}');
+                } catch (linkError) {
+                  console.error('Error clicking ${link}:', linkError);
+                }
+              }
+
+            } catch (error) {
+              console.error('Error fetching latest email or clicking links:', error);
+            }
+          };
+
+          getLatestEmailAndClickLinks();
+```
+
+### Using `exec` pwsh commands in a test file
+
+In a test file, you can use the `pwsh` context directly:
+
+```yaml calculator.yaml highlight={5-8} [expandable]
+version: 6.0.0
+steps:
+  - prompt: launch a calculator
+    commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          start /B calc.exe
+          timeout /t 5
+      - command: wait-for-text
+        text: "calculator"
+        timeout: 30000
+  - prompt: performing the operation 2 + 2 = on the calculator that is opened
+    commands:
+      - command: focus-application
+        name: galculator
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: plus button on the calculator
+        action: click
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: equals button on the calculator
+        action: click
+```
+
+### One more option
+
+You can also save reusable snippets (like launching the calculator) to be inserted into a script later with the [`run`](/commands/run) command. That version would look something like this:
+
+```yaml snippets/launch-calculator.yaml [expandable]
+version: 6.0.0
+steps:
+  - prompt: launch a calculator
+    commands:
+      - command: exec
+        lang: pwsh
+        code: |
+        start /B calc.exe
+          timeout /t 5
+      - command: wait-for-text
+        text: "calculator"
+        timeout: 30000
+```
+
+Then in the test:
+
+```yaml calculator.yaml highlight={5-6} [expandable]
+version: 6.0.0
+steps:
+  - prompt: launch a calculator
+    commands:
+      - command: run
+        file: snippets/launch-calculator.yaml
+  - prompt: performing the operation 2 + 2 = on the calculator that is opened
+    commands:
+      - command: focus-application
+        name: galculator
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: plus button on the calculator
+        action: click
+      - command: hover-image
+        description: button with number 2 on the calculator
+        action: click
+      - command: hover-image
+        description: equals button on the calculator
+        action: click
+```
+
+### Don't try to run `js` within a test field
+
+This example will fail at runtime, so don't try to execute `js` context directly in a test file. Remember - use this in `prerun` to setup your test!
+
+```yaml badtestfile.yaml [expandable]
+version: 6.0.0
+commands:
+      - command: exec
+        lang: pwsh
+        code: |
+          npm install -g axios json2csv
+  - prompt: fetch user data from API
+    commands:
+      - command: exec
+        output: user
+        lang: js
+        code: |
+          const axios = require('axios');
+          const { Parser } = require('json2csv');
+          const fs = require('fs');
+
+          const response = await axios.get('https://jsonplaceholder.typicode.com/users');
+          const parser = new Parser();
+          const csv = parser.parse(response.data);
+          fs.writeFileSync('users.csv', csv);
+          const user = response.data[0].name;
+          result = user;
+          console.log('username', user);
+    ...
+```
+
+This example will produce errors in the TestDriver output or CLI since the runner won't have access to the Node.js VM context.

package/docs/v6/commands/focus-application.mdx

@@ -0,0 +1,44 @@
+---
+title: "focus-application"
+sidebarTitle: "focus-application"
+description: "Bring a specific application window to the foreground."
+icon: "window-restore"
+mode: "wide"
+---
+
+## Description
+
+The `focus-application` command displays a specific application window to the foreground. This ensures that subsequent commands interact with the correct application during a test.
+
+## Arguments
+
+| Argument | Type     | Description                                                                             |
+| -------- | -------- | --------------------------------------------------------------------------------------- |
+| `name`   | `string` | The name of the application to focus. This should match the application's display name. |
+
+## Example usage
+
+```yaml
+command: focus-application
+name: Google Chrome
+```
+
+## Protips
+
+- Ensure the application name matches the exact name displayed in your operating system's task manager.
+- Use this command at the start of a test or before interacting with an application to avoid focus-related issues.
+
+<Note>
+If the specified application isn't running, the command will fail. Ensure the application is open before using this command.
+For example, to launch chrome try using the exec command:
+
+```yaml
+- command: exec
+  lang: pwsh
+  code: start chrome
+```
+
+</Note>
+## Notes
+
+- The `focus-application` command is useful for multi-application workflows where you need to switch between different apps during a test.