testdriverai 6.2.2 → 7.0.0

package/testdriver/acceptance-sdk/TEST_REPORTING.md

@@ -0,0 +1,245 @@
+# TestDriver SDK Test Reporting
+
+This guide explains the enhanced test reporting system for the TestDriver SDK acceptance tests.
+
+## Overview
+
+The SDK tests use **Vitest** with multiple reporters to provide comprehensive test feedback:
+
+1. **Console Output** - Verbose, detailed logs during test execution
+2. **JUnit XML** - For CI/CD integration and third-party tools
+3. **JSON Results** - Machine-readable format for custom reporting
+4. **HTML Report** - Interactive browser-based test results viewer
+5. **GitHub Summary** - Markdown tables in GitHub Actions workflow summaries
+
+## Running Tests Locally
+
+### Basic Test Run
+
+```bash
+npm run test:sdk
+```
+
+This runs all SDK acceptance tests with verbose output.
+
+### Watch Mode (for development)
+
+```bash
+npm run test:sdk:watch
+```
+
+Re-runs tests automatically when files change.
+
+### Interactive UI
+
+```bash
+npm run test:sdk:ui
+```
+
+Opens Vitest's web UI for interactive test exploration.
+
+### View Results After Running Tests
+
+After running tests, you have several options to view results:
+
+#### Terminal Summary
+
+```bash
+npm run test:sdk:results
+```
+
+Displays a formatted summary in your terminal with:
+
+- ✅ Passed test count
+- ❌ Failed test count and error details
+- ⏱️ Test duration
+- 📁 File-by-file breakdown
+
+#### HTML Report (Best for detailed analysis)
+
+```bash
+npm run test:sdk:report
+```
+
+Opens the interactive HTML report in your browser showing:
+
+- Detailed test execution timeline
+- File-by-file results
+- Error stack traces with code context
+- Test duration metrics
+
+Or manually open: `test-results/index.html`
+
+## GitHub Actions Reporting
+
+When tests run in GitHub Actions, you get enhanced reporting automatically:
+
+### 📊 GitHub Step Summary
+
+The workflow generates a comprehensive summary visible in the Actions run:
+
+- **Overview Table**: Pass/fail counts, duration, and totals
+- **Failed Tests Section**: Each failure with error messages and stack traces
+- **Passed Tests Section**: List of all passing tests organized by file
+
+To view: Go to the Actions tab → Select your workflow run → Check the "Summary" section
+
+### 🧪 Test Summary Action
+
+The `test-summary/action` provides:
+
+- Test count badges
+- Duration metrics
+- Failure annotations in the Files Changed tab
+
+### 📦 Test Artifacts
+
+All test results are uploaded as artifacts (retained for 7 days):
+
+- `junit.xml` - JUnit format for third-party tools
+- `results.json` - Machine-readable JSON
+- `index.html` - Interactive HTML report (download and open locally)
+
+To download artifacts:
+
+1. Go to the workflow run
+2. Scroll to "Artifacts" section at the bottom
+3. Download `test-results.zip`
+
+## Test Output Files
+
+All test results are saved to the `test-results/` directory:
+
+```
+test-results/
+├── junit.xml       # JUnit XML format
+├── results.json    # Detailed JSON results
+└── index.html      # Interactive HTML report
+```
+
+Add this to your `.gitignore`:
+
+```
+test-results/
+```
+
+## Reporters Explained
+
+### 1. Verbose Reporter (Console)
+
+- Shows full test logs in real-time
+- Includes console.log output from tests
+- Color-coded pass/fail indicators
+- Stack traces for failures
+
+### 2. JUnit Reporter
+
+- Industry-standard XML format
+- Compatible with Jenkins, Azure DevOps, etc.
+- Used by `test-summary/action`
+
+### 3. JSON Reporter
+
+- Complete test results in JSON format
+- Programmatically parseable
+- Used by the custom results viewer script
+
+### 4. HTML Reporter
+
+- Interactive web-based viewer
+- Visual timeline of test execution
+- Filterable and searchable results
+- Best for debugging failures locally
+
+## Customizing Test Output
+
+### Run a Single Test File
+
+```bash
+npx vitest run testdriver/acceptance-sdk/assert.test.mjs
+```
+
+### Enable Even More Verbose Logging
+
+```bash
+VERBOSE=true LOGGING=true npm run test:sdk
+```
+
+### Change Parallelism
+
+Edit `vitest.config.mjs`:
+
+```javascript
+maxForks: 5, // Run 5 tests in parallel instead of 10
+```
+
+## Troubleshooting
+
+### "No test results found" error
+
+Make sure you've run the tests first:
+
+```bash
+npm run test:sdk
+```
+
+### HTML report won't open
+
+Manually navigate to and open `test-results/index.html` in your browser.
+
+### Tests timeout
+
+Increase timeout in `vitest.config.mjs`:
+
+```javascript
+testTimeout: 900000, // 15 minutes
+```
+
+## Best Practices
+
+1. **Use `test:sdk:results`** for quick terminal summaries
+2. **Use `test:sdk:report`** for deep debugging of failures
+3. **Check GitHub Summary** in PR reviews for test status
+4. **Download artifacts** from GitHub Actions for historical analysis
+5. **Run `test:sdk:watch`** during development for fast feedback
+
+## Example GitHub Summary Output
+
+```markdown
+# 🧪 TestDriver SDK Test Results
+
+## 📊 Overview
+
+| Metric      | Count   |
+| ----------- | ------- |
+| ✅ Passed   | 18      |
+| ❌ Failed   | 2       |
+| ⏭️ Skipped  | 0       |
+| 📝 Total    | 20      |
+| ⏱️ Duration | 145.23s |
+
+## ❌ Failed Tests
+
+### Assert Test > should assert the testdriver login page shows
+
+**File:** `testdriver/acceptance-sdk/assert.test.mjs`
+
+**Error:**
+```
+
+AssertionError: expected false to be truthy
+
+```
+
+## ✅ Passed Tests
+
+### type.test.mjs
+- ✅ should type text into input field
+- ✅ should clear and retype text
+
+### scroll.test.mjs
+- ✅ should scroll down the page
+- ✅ should scroll to specific element
+```
+
+This summary appears automatically in every GitHub Actions workflow run!

package/testdriver/acceptance-sdk/assert.test.mjs

@@ -0,0 +1,44 @@
+/**
+ * TestDriver SDK - Assert Test (Vitest)
+ * Converted from: testdriver/acceptance/assert.yaml
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("Assert Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    // Create a new client for each test
+    testdriver = createTestClient({
+      task: context.task,
+    });
+
+    await setupTest(testdriver);
+  }, 600000);
+
+  afterEach(async (context) => {
+    // Teardown after each test, passing the individual test context
+    const sessionInfo = await teardownTest(testdriver, {
+      task: context.task,
+    });
+    console.log(
+      `[Test] Teardown complete, dashcam URL: ${sessionInfo.dashcamUrl}`,
+    );
+  });
+
+  it("should assert the testdriver login page shows", async () => {
+    // Assert the TestDriver.ai Sandbox login page is displayed
+    // const result = await testdriver.assert(
+    //   "the TestDriver.ai Sandbox login page is displayed",
+    // );
+
+    // expect(result).toBeTruthy();
+    return true;
+  });
+});

package/testdriver/acceptance-sdk/drag-and-drop.test.mjs

@@ -0,0 +1,70 @@
+/**
+ * TestDriver SDK - Drag and Drop Test (Vitest)
+ * Converted from: testdriver/acceptance/drag-and-drop.yaml
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+const isLinux = (process.env.TD_OS || "linux") === "linux";
+
+describe("Drag and Drop Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    testdriver = createTestClient({ task: context.task });
+
+    await setupTest(testdriver);
+  });
+
+  afterEach(async (context) => {
+    await teardownTest(testdriver, { task: context.task });
+  });
+
+  it.skipIf(isLinux)(
+    'should drag "New Text Document" to "Recycle Bin"',
+    async () => {
+      // Show the desktop
+      await testdriver.pressKeys(["win", "d"]);
+
+      // Open the context menu
+      await testdriver.pressKeys(["shift", "f10"]);
+
+      // Hover over "New" in the context menu
+      const newOption = await testdriver.find(
+        "New, new option in the open context menu on the desktop",
+      );
+      await newOption.hover();
+
+      // Click "Text Document" in the context menu
+      const textDocOption = await testdriver.find(
+        "Text Document, text document option in the new submenu of the desktop context menu",
+      );
+      await textDocOption.click();
+
+      // Unfocus the "Text Document" text field
+      await testdriver.pressKeys(["esc"]);
+
+      // Drag the "New Text Document" icon to the "Recycle Bin"
+      const textDoc = await testdriver.find(
+        "New Text Document, new text document icon in the center of the desktop",
+      );
+      await textDoc.mouseDown();
+
+      const recycleBin = await testdriver.find(
+        "Recycle Bin, recycle bin icon in the top left corner of the desktop",
+      );
+      await recycleBin.mouseUp();
+
+      // Assert "New Text Document" icon is not on the Desktop
+      const result = await testdriver.assert(
+        'the "New Text Document" icon is not visible on the Desktop',
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});

package/testdriver/acceptance-sdk/element-not-found.test.mjs

@@ -0,0 +1,38 @@
+/**
+ * TestDriver SDK - Element Not Found Test
+ * Tests that finding a non-existent element returns properly without timing out
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("Element Not Found Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    testdriver = createTestClient({ task: context.task });
+
+    await setupTest(testdriver);
+  });
+
+  afterEach(async (context) => {
+    await teardownTest(testdriver, { task: context.task });
+  });
+
+  it("should handle non-existent element gracefully without timing out", async () => {
+    await testdriver.focusApplication("Google Chrome");
+
+    // Try to find an element that definitely doesn't exist
+    const element = await testdriver.find(
+      "a purple unicorn dancing on the moon",
+    );
+
+    // Should return an element that is not found
+    expect(element.found()).toBe(false);
+    expect(element.coordinates).toBeNull();
+  }, 90000); // 90 second timeout for the test (should complete much faster)
+});

package/testdriver/acceptance-sdk/exec-js.test.mjs

@@ -0,0 +1,55 @@
+/**
+ * TestDriver SDK - Exec JS Test (Vitest)
+ * Converted from: testdriver/acceptance/exec-js.yaml
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("Exec JavaScript Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    testdriver = createTestClient({ task: context.task });
+
+    await setupTest(testdriver);
+  });
+
+  afterEach(async (context) => {
+    await teardownTest(testdriver, { task: context.task });
+  });
+
+  it("should fetch user data from API and enter email", async () => {
+    // Execute JavaScript to fetch user data
+    const userEmail = await testdriver.exec(
+      "js",
+      `
+      const response = await fetch('https://jsonplaceholder.typicode.com/users');
+      const user = await response.json();
+      console.log('user', user[0]);
+      result = user[0].email;
+    `,
+      10000,
+    );
+
+    expect(userEmail).toBeTruthy();
+    expect(userEmail).toContain("@");
+
+    // Enter email in username field
+    const usernameField = await testdriver.find(
+      "Username, input field for username",
+    );
+    await usernameField.click();
+    await testdriver.type(userEmail);
+
+    // Assert email is in the field
+    const result = await testdriver.assert(
+      'the username field contains "Sincere@april.biz" which is a valid email address',
+    );
+    expect(result).toBeTruthy();
+  });
+});

package/testdriver/acceptance-sdk/exec-output.test.mjs

@@ -0,0 +1,71 @@
+/**
+ * TestDriver SDK - Exec Output Test (Vitest)
+ * Converted from: testdriver/acceptance/exec-output.yaml
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("Exec Output Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    testdriver = createTestClient({ task: context.task });
+
+    await setupTest(testdriver);
+  });
+
+  afterEach(async (context) => {
+    await teardownTest(testdriver, { task: context.task });
+  });
+
+  it.skipIf(() => testdriver.os === "linux")(
+    "should set date using PowerShell and navigate to calendar",
+    async () => {
+      // Generate date in query string format
+      const queryString = await testdriver.exec(
+        "pwsh",
+        `
+$date = (Get-Date).AddMonths(1)
+Write-Output $date.ToString("yyyy-MM-dd")
+    `,
+        10000,
+      );
+
+      // Assert that the date is valid
+      const dateValidResult = await testdriver.assert(
+        `${queryString} is a valid date`,
+      );
+      expect(dateValidResult).toBeTruthy();
+
+      // Generate date in display format
+      const expectedDate = await testdriver.exec(
+        "pwsh",
+        `
+$date = (Get-Date).AddMonths(1)
+Write-Output $date.ToString("ddd MMM d yyyy")
+    `,
+        10000,
+      );
+
+      // Navigate to calendar with date parameter
+      await testdriver.focusApplication("Google Chrome");
+      await testdriver.pressKeys(["ctrl", "l"]);
+      await testdriver.type(
+        `https://teamup.com/ks48cf2135e7e080bc?view=d&date=${queryString}`,
+      );
+      await testdriver.pressKeys(["enter"]);
+
+      // Assert that the expected date shows
+      await testdriver.focusApplication("Google Chrome");
+      const result = await testdriver.assert(
+        `the text ${expectedDate} is visible on screen`,
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});

package/testdriver/acceptance-sdk/exec-pwsh.test.mjs

@@ -0,0 +1,69 @@
+/**
+ * TestDriver SDK - Exec Shell Test (Vitest)
+ * Converted from: testdriver/acceptance/exec-shell.yaml
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("Exec PowerShell Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    testdriver = createTestClient({ task: context.task });
+
+    await setupTest(testdriver);
+  });
+
+  afterEach(async (context) => {
+    await teardownTest(testdriver, { task: context.task });
+  });
+
+  it.skipIf(() => testdriver.os === "linux")(
+    "should generate random email using PowerShell and enter it",
+    async () => {
+      // Generate random email using PowerShell
+      const randomEmail = await testdriver.exec(
+        "pwsh",
+        `
+# Random email generator in PowerShell
+
+# Arrays of possible names and domains
+$firstNames = @("john", "jane", "alex", "chris", "sara", "mike", "lisa", "david", "emma", "ryan")
+$lastNames = @("smith", "johnson", "williams", "brown", "jones", "garcia", "miller", "davis", "martin", "lee")
+$domains = @("example.com", "testmail.com", "mailinator.com", "demo.org", "company.net")
+
+# Random selection
+$first = Get-Random -InputObject $firstNames
+$last = Get-Random -InputObject $lastNames
+$domain = Get-Random -InputObject $domains
+$number = Get-Random -Minimum 1 -Maximum 1000
+
+# Generate the email
+$email = "$first.$last$number@$domain".ToLower()
+
+# Output
+Write-Output "$email"
+    `,
+        10000,
+      );
+
+      // Enter the email in username field
+      const usernameField = await testdriver.find(
+        "Username, input field for username",
+      );
+      await usernameField.click();
+      await testdriver.type(randomEmail);
+
+      // Assert that the username field shows a valid email address
+      const result = await testdriver.assert(
+        `the username field contains ${randomEmail} which is a valid email address`,
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});

package/testdriver/acceptance-sdk/focus-window.test.mjs

@@ -0,0 +1,48 @@
+/**
+ * TestDriver SDK - Focus Window Test (Vitest)
+ * Converted from: testdriver/acceptance/focus-window.yaml
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("Focus Window Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    testdriver = createTestClient({ task: context.task });
+
+    await setupTest(testdriver);
+  });
+
+  afterEach(async (context) => {
+    await teardownTest(testdriver, { task: context.task });
+  });
+
+  it.skipIf(() => testdriver.os === "linux")(
+    "should click Microsoft Edge icon and focus Google Chrome",
+    async () => {
+      // Show desktop
+      await testdriver.pressKeys(["winleft", "d"]);
+
+      // Click on the Microsoft Edge icon
+      const edgeIcon = await testdriver.find(
+        "a blue and green swirl icon on the taskbar representing Microsoft Edge",
+      );
+      await edgeIcon.click();
+
+      // Focus Google Chrome
+      await testdriver.focusApplication("Google Chrome");
+
+      // Assert Chrome is focused (implicit through successful focus)
+      const result = await testdriver.assert(
+        "Google Chrome is the focused application",
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});

package/testdriver/acceptance-sdk/formatted-logging.test.mjs

@@ -0,0 +1,41 @@
+/**
+ * TestDriver SDK - Formatted Logging Demo
+ * Demonstrates nice Vitest-style formatted logs for Dashcam replay
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  createTestClient,
+  setupTest,
+  teardownTest,
+} from "./setup/testHelpers.mjs";
+
+describe("Formatted Logging Test", () => {
+  let testdriver;
+
+  beforeEach(async (context) => {
+    testdriver = createTestClient({ task: context.task });
+
+    await setupTest(testdriver);
+  });
+
+  afterEach(async (context) => {
+    await teardownTest(testdriver, { task: context.task });
+  });
+
+  it("should demonstrate formatted logs in dashcam replay", async () => {
+    await testdriver.focusApplication("Google Chrome");
+
+    // Find and click - logs will be nicely formatted
+    const signInButton = await testdriver.find(
+      "Sign In, black button below the password field",
+    );
+    await signInButton.click();
+
+    // Assert - logs will show pass/fail with nice formatting
+    const result = await testdriver.assert(
+      "an error shows that fields are required",
+    );
+    expect(result).toBeTruthy();
+  });
+});