testdriverai 6.0.23 → 6.0.24-canary.fffd04a.0

package/.github/workflows/acceptance-tests.yml

@@ -14,7 +14,11 @@ jobs:
   gather-test-files:
     name: Gather Test Files
     runs-on: ubuntu-latest
-    if: github.event.pull_request.base.ref == 'main' || github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch'
+    if: >
+      (github.event_name == 'workflow_dispatch') ||
+      (github.ref == 'refs/heads/main') ||
+      (github.event_name == 'pull_request_review' && 
+       (github.event.review.state == 'approved' || github.event.review.state == 'changes_requested'))
     outputs:
       test_files: ${{ steps.test_list.outputs.files }}
     steps:

package/docs/action/ami.mdx

@@ -6,9 +6,10 @@ icon: "image"
 ---
 
 <Tip>
-  This guide is specifically for **Enterprise Plan customers** who have access
-  to the shared TestDriver AMI. If you are on a different plan, please contact
-  your account manager for more information.
+  This guide is specifically for [Enterprise Plan
+  customers](/account/enterprise) who have access to the shared TestDriver AMI.
+  If you are on a different plan, please contact your account manager for more
+  information.
 </Tip>
 
 <Steps>

package/docs/action/performance.mdx

@@ -28,15 +28,28 @@ Parallel testing allows you to split your test actions into multiple files and r
   manageable pieces. To run multiple separate tests, use a test matrix strategy.
 </Warning>
 
-Example:
+#### Example
 
-```yaml
-strategy:
-  matrix:
-    test_file:
-      - tests/login.yaml
-      - tests/search.yaml
-      - tests/cart.yaml
+```yaml .github/workflows/testdriver.yaml focus={3-9, 14}
+run tests:
+  name: Run tests in parallel
+  strategy:
+    matrix:
+      test_file:
+        - tests/login.yaml
+        - tests/search.yaml
+        - tests/cart.yaml
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Run the tests
+          run: npx testdriverai@latest run testdriver/${{ matrix.test_file }}
+          env:
+            FORCE_COLOR: 3
+            TD_API_KEY: ${{ secrets.TD_API_KEY }}
+            TD_WEBSITE: https://testdriver-sandbox.vercel.app
+            TD_THIS_FILE: ${{ matrix.test }}
 ```
 
 ---
@@ -55,6 +68,11 @@ action: click
 method: turbo
 ```
 
+<Note>
+  `turbo` is the default method that is currently implemented. The other method
+  being `ai`
+</Note>
+
 ---
 
 ### 3. Use `async` asserts

package/docs/action/secrets.mdx

@@ -25,8 +25,8 @@ To securely use secrets in your TestDriver test files, replace hardcoded values
 
 ### Example test file
 
-```yaml
-version: 4.1.35
+```yaml login.yaml highlight={11, 12, 21, 22}
+version: 6.0.0
 steps:
   - prompt: sign in with username and password
     commands:
@@ -48,6 +48,10 @@ steps:
         action: click
       - command: type
         text: ${TD_PASSWORD}
+      - command: hover-text
+        text: Log In
+        description: log in button
+        action: click
 ```
 
 ---
@@ -69,42 +73,18 @@ For detailed instructions, refer to the [GitHub Docs on using secrets](https://d
 
 When running TestDriver tests via the CLI in GitHub Actions, supply your secrets in the `env` section of the workflow step.
 
-### Example workflow
-
-```yaml .github/workflows/testdriver.yaml
-name: TestDriver Test
-
-permissions:
-  actions: read
-  contents: read
-  statuses: write
-  pull-requests: write
-
-jobs:
-  test:
-    name: "TestDriver"
-    steps:
-      - name: Check out repository
-        uses: actions/checkout@v4
-
-      - name: Run TestDriver
-        run: npx testdriverai@latest run tests/signin.yaml --headless
-        env:
-          TD_API_KEY: ${{ secrets.TD_API_KEY }}
-          TD_USERNAME: ${{ secrets.TD_USERNAME }}
-          TD_PASSWORD: ${{ secrets.TD_PASSWORD }}
-```
-
----
+<Card title="Adding secrets to CI" icon="link" href="/guide/authentication">
+  Check out our authentication page to see how to add secrets in your CI
+  workflow.
+</Card>
 
 ## Best practices
 
 - **Use Descriptive Names**: Name your secrets clearly (for example, `TD_USERNAME`, `TD_PASSWORD`) to make them easy to identify.
 - **Rotate Secrets Regularly**: Update your secrets periodically to enhance security.
 - **Limit Access**: Only provide access to secrets for workflows and team members that require them.
-- **Mask Secrets**: Ensure all secrets are masked in logs by using the `TD_` prefix.
 
----
+## <Note>To ensure all secrets are masked in logs, use the `TD_` prefix.</Note>
 
 ## Notes
 

package/docs/apps/chrome-extensions.mdx

@@ -29,7 +29,7 @@ TestDriver provides a powerful solution for testing Chrome extensions. With our
 
 ## Getting started
 
-You can test a preinstalled extension or test by installing an extension. If the extension is in development mode, turn this on by checking the "Developer mode" checkbox in the Chrome extensions page. From there you can load the unpacked extension by selecting the folder where your extension is located. To download the files onto the VM if you are performing a test on cloud hosted runners, see the [Lifecycle](/guide/lifecycle-prerun) and [Prerun](/action/pre-post-scripts) docs.
+You can test a preinstalled extension or test by installing an extension. If the extension is in development mode, turn this on by checking the "Developer mode" checkbox in the Chrome extensions page. From there you can load the unpacked extension by selecting the folder where your extension is located. To download the files onto the VM if you are performing a test on cloud hosted runners, see the [Lifecycle](/guide/lifecycle) docs.
 
 ## Sample test steps
 

package/docs/apps/desktop-apps.mdx

@@ -46,28 +46,29 @@ TestDriver supports a wide range of desktop applications, including:
 
 ## Installation
 
-To use TestDriver with desktop applications, no special setup is needed. Simply run the local agent to interact with applications on your local machine, or add instructions to `prerun.yaml` to tell the runner to launch or install the application. Here is an example that performs a calculation on a calculator app:
+To use TestDriver with desktop applications, no special setup is needed. Simply run the local agent to interact with applications on your local machine, or add instructions to `provision.yaml` to tell the runner to launch or install the application. Here is an example that performs a calculation on a calculator app:
 
-```yaml
-version: 5.5.5
+```yaml testdriver/lifecycle/provision.yaml
+version: 6.0.0
 steps:
   - prompt: launch a calculator
     commands:
       - command: exec
         lang: pwsh
-        code: start /B calc.exe
+        code: |
+          start /B calc.exe
           timeout /t 5
       - command: wait-for-text
         text: "calculator"
         timeout: 30000
 ```
 
-A `calculator_test.yaml` file might look like this:
+A `calculator.yaml` file might look like this:
 
-```yaml
-version: 5.5.5
+```yaml calculator.yaml
+version: 6.0.0
 steps:
-  - prompt: /try performing the operation 2 + 2 = on the calculator that is opened
+  - prompt: perform the operation 2 + 2 = on the calculator that is opened
     commands:
       - command: focus-application
         name: galculator
@@ -83,6 +84,8 @@ steps:
       - command: hover-image
         description: equals button on the calculator
         action: click
+      - command: assert
+        expected: 4 is displayed as the result
 ```
 
 ## Conclusion

package/docs/apps/static-websites.mdx

@@ -28,27 +28,27 @@ To get started with testing static websites using TestDriver, follow these steps
 1. [**Follow our quick start guide to set up your TestDriver account.**](/getting-started/setup)
 2. **Create a new test project** in your TestDriver dashboard.
 3. **Add your static website URL** to the test settings.
-   It looks like this in the `prerun.yaml` file:
+   It looks like this in the `provision.yaml` file:
 
-```yaml
-version: 5.1.1
+```yaml testdriver/lifecycle/provision.yaml
+version: 6.0.0
 steps:
   - prompt: launch chrome
     commands:
       - command: exec
         lang: pwsh
-        code:
+        code: |
           Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "${TD_WEBSITE}"
           exit
       - command: wait-for-text
-        text: "Google Chrome"
+        text: ${TD_WEBSITE}
         timeout: 30000
 ```
 
-Set the `TD_WEBSITE` environment variable to the URL of your static website. This will allow TestDriver to launch the browser and navigate to your website. You can do this by adding this `prerun.yaml` file to your project or by simply using:
+Set the `TD_WEBSITE` environment variable to the URL of your static website. This will allow TestDriver to launch the browser and navigate to your website. You can do this by adding this `provision.yaml` file to your project or by simply using:
 
 ```bash
 npx testdriverai@latest init
 ```
 
-Running init will create the `prerun.yaml` file for you and ask you which website to load with your test.
+Running init will create the `provision.yaml` file for you and ask you which website to load with your test.

package/docs/bugs/jira.mdx

@@ -42,9 +42,9 @@ This guide explains how to integrate the **TestDriver CLI** with the **Jira GitH
 
 Here's a complete workflow that integrates TestDriver and Jira:
 
-### Workflow File: `.github/workflows/testdriver-jira.yaml`
+### Workflow File:
 
-```yaml [expandable]
+```yaml .github/workflows/testdriver-jira.yaml [expandable]
 name: TestDriver with Jira Integration
 
 on:
@@ -69,10 +69,11 @@ jobs:
 
       - name: Run TestDriver
         id: testdriver
+        shell: powershell
         run: |
           $exitCode = 0
           try {
-            npx testdriverai@latest run testdriver/test.yaml --headless --key $env:TD_API_KEY --output="testdriver-output.txt"
+            npx testdriverai@latest run testdriver/test.yaml --summary="testdriver-summary.txt"
           } catch {
             $exitCode = $LASTEXITCODE
           }
@@ -86,22 +87,21 @@ jobs:
           }
         env:
           TD_API_KEY: ${{ secrets.TD_API_KEY }}
-        shell: powershell
 
       - name: Check for Test Failures
         id: check-failures
+        shell: powershell
         run: |
           if ("${{ steps.testdriver.outputs.success }}" -eq "false") {
             echo "TestDriver tests failed."
             echo "failure=true" >> $env:GITHUB_ENV
             
             # Copy the output file for Jira ticket
-            Copy-Item testdriver-output.txt failure-details.txt
+            Copy-Item testdriver-summary.txt failure-details.txt
           } else {
             echo "All tests passed."
             echo "failure=false" >> $env:GITHUB_ENV
           }
-        shell: powershell
 
       - name: Upload test results
         if: always()
@@ -109,7 +109,7 @@ jobs:
         with:
           name: testdriver-results
           path: |
-            testdriver-output.txt
+            testdriver-summary.txt
             failure-details.txt
 
   create-jira-ticket:
@@ -152,15 +152,14 @@ jobs:
 
 ### 1. **Run Tests with TestDriver CLI**
 
-The TestDriver CLI runs your tests in headless mode using the `--headless` flag. The `--output` flag specifies where to save the test results for later use in Jira tickets.
+This step runs the TestDriver CLI with the `testdriver/test.yaml` file.
 
-```yaml
+```yaml testdriver/test.yaml
 - name: Run TestDriver
   id: testdriver
-  run: npx testdriverai@latest run testdriver/test.yaml --headless --key $env:TD_API_KEY --output="testdriver-output.txt"
+  run: npx testdriverai@latest run testdriver/test.yaml
   env:
     TD_API_KEY: ${{ secrets.TD_API_KEY }}
-  shell: powershell
 ```
 
 ---
@@ -169,7 +168,7 @@ The TestDriver CLI runs your tests in headless mode using the `--headless` flag.
 
 This step checks the exit code and success status from the TestDriver CLI run. If the tests failed, it sets an environment variable (`failure=true`) to trigger the Jira ticket creation step and prepares the output file for the Jira ticket.
 
-```yaml
+```yaml testdriver/test.yaml
 - name: Check for Test Failures
   id: check-failures
   run: |
@@ -195,7 +194,7 @@ If any tests failed, the `create-jira-ticket` job uses the `atlassian/gajira-cre
 - **Summary**: A brief description of the failure with repository information.
 - **Description**: Detailed information about the failure, including repository details, workflow run links, and the captured test output.
 
-```yaml
+```yaml testdriver/test.yaml
 - name: Create Jira Ticket
   uses: atlassian/gajira-create@v3
   with:
@@ -231,65 +230,3 @@ Add the following secrets to your GitHub repository:
 4. **`JIRA_USERNAME`**: Your Jira account email.
 
 ---
-
-## Lifecycle files
-
-TestDriver supports lifecycle files to customize test execution at different phases. These files replace the previous `prerun` parameter from the GitHub Action and provide more comprehensive control over the test environment.
-
-The main lifecycle files are:
-
-- **`lifecycle/provision.yaml`**: Executed when a new sandbox is created
-- **`lifecycle/prerun.yaml`**: Executed before tests run (replaces the old `prerun` parameter)
-- **`lifecycle/postrun.yaml`**: Executed after tests complete
-
-<Note>
-  These lifecycle files should be placed in your repository's `lifecycle/`
-  directory and will be automatically executed by TestDriver during the
-  appropriate phases.
-</Note>
-
-For detailed information about lifecycle files, including complete examples and best practices, see the [Lifecycle Files guide](/guide/lifecycle).
-
----
-
-## Example Jira ticket
-
-### Summary:
-
-`TestDriver Test Failure in mycompany/myproject`
-
-### Description:
-
-```
-A test failure occurred during the TestDriver workflow.
-
-**Repository**: mycompany/myproject
-**Branch**: refs/heads/main
-**Commit**: abc123def456
-**Workflow Run**: https://github.com/mycompany/myproject/actions/runs/123456789
-
-**Failure Details**:
-```
-
-TestDriver CLI execution failed
-Error: Test 'Login Test' failed - element not found
-Exit code: 1
-
-```
-
-Please investigate the issue and resolve it promptly.
-
-```
-
----
-
-## Benefits of this workflow
-
-1. **Automated Issue Tracking**: Automatically creates Jira tickets for test failures, ensuring no issues are overlooked.
-2. **Detailed Context**: Includes test output, repository information, and workflow run links in the Jira ticket for easier debugging.
-3. **Streamlined Workflow**: Integrates testing and issue tracking into a single automated pipeline.
-4. **Flexible Lifecycle Management**: Supports custom setup, provisioning, and cleanup through lifecycle files.
-
----
-
-By combining TestDriver CLI with Jira GitHub Actions, you can automate the process of tracking test failures, improving collaboration and ensuring faster resolution of issues.

package/docs/cli/overview.mdx

@@ -15,20 +15,20 @@ npx testdriverai@latest <command> [options]
 
 ## Available commands
 
-| Command                | Description                                                  |
-| ---------------------- | ------------------------------------------------------------ |
+|        Command         | Description                                                  |
+| :--------------------: | :----------------------------------------------------------- |
 | [`run`](/commands/run) | Executes a TestDriver test.                                  |
-| `edit`                 | Launch interactive mode.                                     |
-| `help`                 | Displays help information for the CLI or a specific command. |
+| [`edit`](/commands/edit)         | Launch interactive mode.                                     |
+| [`help`](/commands/help)         | Displays help information for the CLI or a specific command. |
 
 ## Available Flags
 
 | Flag                | Description                                                                                |
-| ------------------- | ------------------------------------------------------------------------------------------ |
+| :------------------ | :----------------------------------------------------------------------------------------- |
 | `--heal`            | Launch exploratory mode and attemp to recover if an error or failing state is encountered. |
 | `--write`           | Ovewrite test file with new commands resulting from agentic testing                        |
 | `--headless`        | Run test without opening a browser window (useful for CI/CD environments)                  |
-| `--new-sandbox`     | Create a new sandbox environment for the test run.                                         |
+| `--new`             | Create a new sandbox environment for the test run.                                         |
 | `--summary=<value>` | Output file where AI summary should be saved.                                              |
 
 ## Example usage

package/docs/commands/exec.mdx

@@ -18,29 +18,29 @@ The `exec` command allows you to execute custom Node.js scripts within your Test
 
 ## Arguments
 
-| Argument | Type     | Description                                                                                                                                                                        |
-| -------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `lang`   | `string` | The language of the script to execute. Supported values are `shell` and `js`.                                                                                                      |
+| Argument |   Type   | Description                                                                                                                                                                        |
+| :------: | :------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+|  `lang`  | `string` | The language of the script to execute. Supported values are `pwsh` and `js`.                                                                                                       |
 | `output` | `string` | The variable name to store the result of the script. This variable can be accessed as `${OUTPUT.<var>}` in future steps.                                                           |
-| `silent` | `string` | Defaults to `false`. The command will print the output of the script. This is useful for suppressing unnecessary or private output in the test logs. This is useful for debugging. |
-| `code`   | `string` | The script to execute on Windows systems. For `js`, the script must define the output as `result`.                                                                                 |
+|  `code`  | `string` | The script to execute on Windows systems. For `js`, the script must define the output as `result`.                                                                                 |
+| `silent` | `string` | Defaults to `false`. The command will print the output of the script. This is useful for suppressing unnecessary or private output in the test logs and it's useful for debugging. |
 
 ## Example usage
 
 This example demonstrates how to use the `exec` command to generate a TOTP (Time-based One-Time Password) using the `totp-generator` library.
 
-```yaml
-version: 5.3.8
+```yaml otp-generator.yaml
+version: 6.0.0
 steps:
   - commands:
       - command: exec
         lang: pwsh
-        linux: |
+        code: |
           npm install totp-generator
       - command: exec
         lang: js
         output: totp
-        linux: |
+        code: |
           const { TOTP } = require("totp-generator");
           let otp = TOTP.generate("JBSWY3DPEB3W64TMMQQQ").otp;
           console.log(otp);
@@ -51,19 +51,16 @@ steps:
 
 ## Additional details
 
-- The `exec` command now takes a `lang` argument and supports different operating systems (`linux`, `mac`, and `windows`).
-- Supported `lang` values are `js` or `shell`:
-  - `js` code is executed in a Node.js VM on the host machine (for example the machine where your CI/CD runs, or your computer if using the local agent).
-  - `shell` code is executed in the shell on target runner (which can be the cloud runner, local sandbox, or local machine, depending on where you run your tests).
-    - Note: You can also use `shell` in [`prerun`](/guide/lifecycle-prerun) scripts to install npm packages if you need them.
-    - Otherwise, the `shell` code can be used within test steps to launch applications or perform simple commands (like writing text to a file on the machine to perform a simple file upload).
-- Code specified in `linux`, `mac`, and `windows` is executed based on the platform of the runner machine.
+- Supported `lang` values are `js` or `pwsh`:
+  - `js` code is executed in a Node.js [VM](https://nodejs.org/api/vm.html) internally on the host machine (for example the machine where your CI/CD runs, or your computer if using the local agent).
+  - `pwsh` code is executed in the shell on target runner (which can be the cloud runner, local sandbox, or local machine, depending on where you run your tests).
+    - **Note:** You can also use `pwsh` in [lifecycle](/guide/lifecycle) scripts to install npm packages if you need them.
+    - Otherwise, the `pwsh` code can be used within test steps to launch applications or perform simple commands (like writing text to a file on the machine to perform a simple file upload).
 - The `output`argument is assigned automatically by setting `result = somestringvalue` in the script you run.
 
 ## Protips
 
 - The `result` variable is already available in your script, overwrite it to store the output as shown in the examples.
-- Node.js code executes in the same context as the calling process on the host machine, using the [VM](https://nodejs.org/api/vm.html) module internally.
 - Do any handling of arrays or nested objects within your `js` script:
   - ✅ `result = users[1].profile.firstName`
   - ✅ `result = data.length > 0 ? data[0].userEmail : 'no user found'` if no data is found the value of output will be `null`
@@ -79,21 +76,19 @@ steps:
 
 ## Ways to use `exec`
 
-Here is an example using both `shell` and `js` contexts within a [`prerun.yaml`](guide/lifecycle-prerun) script:
+Here is an example using both `pwsh` and `js` contexts within a `prerun.yaml` script which creates a temporary email account and automatically clicks links found in received emails.
 
 ```yaml ./lifecycle/prerun.yaml [expandable]
-version: 5.3.8
+version: 6.0.0
 steps:
   - commands:
       - command: exec
         lang: pwsh
-        silent: false
         code: |
           npm install @sendgrid/mail
       - command: exec
         lang: js
         output: accountData
-        silent: false
         code: |
           const Mailjs = require("@cemalgnlts/mailjs");
           const mailjs = new Mailjs();
@@ -103,7 +98,6 @@ steps:
       - command: exec
         lang: js
         output: emailAddress
-        silent: false
         code: |
           const accountData = ${OUTPUT.accountData};
           result = accountData.username
@@ -119,7 +113,6 @@ steps:
     commands:
       - command: exec
         lang: js
-        silent: false
         code: |
           const Mailjs = require("@cemalgnlts/mailjs");
           const { JSDOM } = require('jsdom'); // To parse HTML and extract links
@@ -174,23 +167,24 @@ steps:
           getLatestEmailAndClickLinks();
 ```
 
-### Using `exec` shell commands in a test file
+### Using `exec` pwsh commands in a test file
 
-In a test file, you can use the `shell` context directly:
+In a test file, you can use the `pwsh` context directly:
 
-```yaml testfile.yaml [expandable]
-version: 5.5.5
+```yaml calculator.yaml highlight={5-8} [expandable]
+version: 6.0.0
 steps:
   - prompt: launch a calculator
     commands:
       - command: exec
         lang: pwsh
-        code: start /B calc.exe
+        code: |
+          start /B calc.exe
           timeout /t 5
       - command: wait-for-text
         text: "calculator"
         timeout: 30000
-  - prompt: /try performing the operation 2 + 2 = on the calculator that is opened
+  - prompt: performing the operation 2 + 2 = on the calculator that is opened
     commands:
       - command: focus-application
         name: galculator
@@ -212,14 +206,15 @@ steps:
 
 You can also save reusable snippets (like launching the calculator) to be inserted into a script later with the [`run`](/commands/run) command. That version would look something like this:
 
-```yaml launchcalculator.yaml [expandable]
-version: 5.5.5
+```yaml snippets/launch-calculator.yaml [expandable]
+version: 6.0.0
 steps:
   - prompt: launch a calculator
     commands:
       - command: exec
         lang: pwsh
-        code: start /B calc.exe
+        code: |
+        start /B calc.exe
           timeout /t 5
       - command: wait-for-text
         text: "calculator"
@@ -228,14 +223,14 @@ steps:
 
 Then in the test:
 
-```yaml {5-6} testfile.yaml [expandable]
-version: 5.5.5
+```yaml calculator.yaml highlight={5-6} [expandable]
+version: 6.0.0
 steps:
   - prompt: launch a calculator
     commands:
       - command: run
-        file: ./launchcalculator.yaml
-  - prompt: /try performing the operation 2 + 2 = on the calculator that is opened
+        file: snippets/launch-calculator.yaml
+  - prompt: performing the operation 2 + 2 = on the calculator that is opened
     commands:
       - command: focus-application
         name: galculator
@@ -258,7 +253,7 @@ steps:
 This example will fail at runtime, so don't try to execute `js` context directly in a test file. Remember - use this in `prerun` to setup your test!
 
 ```yaml badtestfile.yaml [expandable]
-version: 5.5.5
+version: 6.0.0
 commands:
       - command: exec
         lang: pwsh

package/docs/commands/focus-application.mdx

@@ -3,11 +3,12 @@ title: "focus-application"
 sidebarTitle: "focus-application"
 description: "Bring a specific application window to the foreground."
 icon: "window-restore"
+mode: "wide"
 ---
 
 ## Description
 
-The `focus-application` command brings a specific application window to the foreground. This ensures that subsequent commands interact with the correct application during a test.
+The `focus-application` command displays a specific application window to the foreground. This ensures that subsequent commands interact with the correct application during a test.
 
 ## Arguments
 
@@ -26,8 +27,18 @@ name: Google Chrome
 
 - Ensure the application name matches the exact name displayed in your operating system's task manager.
 - Use this command at the start of a test or before interacting with an application to avoid focus-related issues.
-- If the specified application isn't running, the command will fail. Ensure the application is open before using this command.
 
+<Note>
+If the specified application isn't running, the command will fail. Ensure the application is open before using this command.
+For example, to launch chrome try using the exec command:
+
+```yaml
+- command: exec
+  lang: pwsh
+  code: start chrome
+```
+
+</Note>
 ## Notes
 
 - The `focus-application` command is useful for multi-application workflows where you need to switch between different apps during a test.