testdriverai 6.2.2 → 7.1.0

package/testdriver/acceptance-sdk/TEST_REPORTING.md

@@ -0,0 +1,245 @@
+# TestDriver SDK Test Reporting
+
+This guide explains the enhanced test reporting system for the TestDriver SDK acceptance tests.
+
+## Overview
+
+The SDK tests use **Vitest** with multiple reporters to provide comprehensive test feedback:
+
+1. **Console Output** - Verbose, detailed logs during test execution
+2. **JUnit XML** - For CI/CD integration and third-party tools
+3. **JSON Results** - Machine-readable format for custom reporting
+4. **HTML Report** - Interactive browser-based test results viewer
+5. **GitHub Summary** - Markdown tables in GitHub Actions workflow summaries
+
+## Running Tests Locally
+
+### Basic Test Run
+
+```bash
+npm run test:sdk
+```
+
+This runs all SDK acceptance tests with verbose output.
+
+### Watch Mode (for development)
+
+```bash
+npm run test:sdk:watch
+```
+
+Re-runs tests automatically when files change.
+
+### Interactive UI
+
+```bash
+npm run test:sdk:ui
+```
+
+Opens Vitest's web UI for interactive test exploration.
+
+### View Results After Running Tests
+
+After running tests, you have several options to view results:
+
+#### Terminal Summary
+
+```bash
+npm run test:sdk:results
+```
+
+Displays a formatted summary in your terminal with:
+
+- ✅ Passed test count
+- ❌ Failed test count and error details
+- ⏱️ Test duration
+- 📁 File-by-file breakdown
+
+#### HTML Report (Best for detailed analysis)
+
+```bash
+npm run test:sdk:report
+```
+
+Opens the interactive HTML report in your browser showing:
+
+- Detailed test execution timeline
+- File-by-file results
+- Error stack traces with code context
+- Test duration metrics
+
+Or manually open: `test-results/index.html`
+
+## GitHub Actions Reporting
+
+When tests run in GitHub Actions, you get enhanced reporting automatically:
+
+### 📊 GitHub Step Summary
+
+The workflow generates a comprehensive summary visible in the Actions run:
+
+- **Overview Table**: Pass/fail counts, duration, and totals
+- **Failed Tests Section**: Each failure with error messages and stack traces
+- **Passed Tests Section**: List of all passing tests organized by file
+
+To view: Go to the Actions tab → Select your workflow run → Check the "Summary" section
+
+### 🧪 Test Summary Action
+
+The `test-summary/action` provides:
+
+- Test count badges
+- Duration metrics
+- Failure annotations in the Files Changed tab
+
+### 📦 Test Artifacts
+
+All test results are uploaded as artifacts (retained for 7 days):
+
+- `junit.xml` - JUnit format for third-party tools
+- `results.json` - Machine-readable JSON
+- `index.html` - Interactive HTML report (download and open locally)
+
+To download artifacts:
+
+1. Go to the workflow run
+2. Scroll to "Artifacts" section at the bottom
+3. Download `test-results.zip`
+
+## Test Output Files
+
+All test results are saved to the `test-results/` directory:
+
+```
+test-results/
+├── junit.xml       # JUnit XML format
+├── results.json    # Detailed JSON results
+└── index.html      # Interactive HTML report
+```
+
+Add this to your `.gitignore`:
+
+```
+test-results/
+```
+
+## Reporters Explained
+
+### 1. Verbose Reporter (Console)
+
+- Shows full test logs in real-time
+- Includes console.log output from tests
+- Color-coded pass/fail indicators
+- Stack traces for failures
+
+### 2. JUnit Reporter
+
+- Industry-standard XML format
+- Compatible with Jenkins, Azure DevOps, etc.
+- Used by `test-summary/action`
+
+### 3. JSON Reporter
+
+- Complete test results in JSON format
+- Programmatically parseable
+- Used by the custom results viewer script
+
+### 4. HTML Reporter
+
+- Interactive web-based viewer
+- Visual timeline of test execution
+- Filterable and searchable results
+- Best for debugging failures locally
+
+## Customizing Test Output
+
+### Run a Single Test File
+
+```bash
+npx vitest run testdriver/acceptance-sdk/assert.test.mjs
+```
+
+### Enable Even More Verbose Logging
+
+```bash
+VERBOSE=true LOGGING=true npm run test:sdk
+```
+
+### Change Parallelism
+
+Edit `vitest.config.mjs`:
+
+```javascript
+maxForks: 5, // Run 5 tests in parallel instead of 10
+```
+
+## Troubleshooting
+
+### "No test results found" error
+
+Make sure you've run the tests first:
+
+```bash
+npm run test:sdk
+```
+
+### HTML report won't open
+
+Manually navigate to and open `test-results/index.html` in your browser.
+
+### Tests timeout
+
+Increase timeout in `vitest.config.mjs`:
+
+```javascript
+testTimeout: 900000, // 15 minutes
+```
+
+## Best Practices
+
+1. **Use `test:sdk:results`** for quick terminal summaries
+2. **Use `test:sdk:report`** for deep debugging of failures
+3. **Check GitHub Summary** in PR reviews for test status
+4. **Download artifacts** from GitHub Actions for historical analysis
+5. **Run `test:sdk:watch`** during development for fast feedback
+
+## Example GitHub Summary Output
+
+```markdown
+# 🧪 TestDriver SDK Test Results
+
+## 📊 Overview
+
+| Metric      | Count   |
+| ----------- | ------- |
+| ✅ Passed   | 18      |
+| ❌ Failed   | 2       |
+| ⏭️ Skipped  | 0       |
+| 📝 Total    | 20      |
+| ⏱️ Duration | 145.23s |
+
+## ❌ Failed Tests
+
+### Assert Test > should assert the testdriver login page shows
+
+**File:** `testdriver/acceptance-sdk/assert.test.mjs`
+
+**Error:**
+```
+
+AssertionError: expected false to be truthy
+
+```
+
+## ✅ Passed Tests
+
+### type.test.mjs
+- ✅ should type text into input field
+- ✅ should clear and retype text
+
+### scroll.test.mjs
+- ✅ should scroll down the page
+- ✅ should scroll to specific element
+```
+
+This summary appears automatically in every GitHub Actions workflow run!

package/testdriver/acceptance-sdk/assert.test.mjs

@@ -0,0 +1,26 @@
+/**
+ * TestDriver SDK - Assert Test (Vitest)
+ * Converted from: testdriver/acceptance/assert.yaml
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+describe("Assert Test", () => {
+  it("should assert the testdriver login page shows", async (context) => {
+    const testdriver = TestDriver(context, { headless: true, newSandbox: true });
+    
+    // provision.chrome() automatically calls ready() and starts dashcam
+    await testdriver.provision.chrome({
+      url: 'http://testdriver-sandbox.vercel.app/login',
+    });
+
+    // Assert the TestDriver.ai Sandbox login page is displayed
+    const result = await testdriver.assert(
+      "the TestDriver.ai Sandbox login page is displayed",
+    );
+
+    expect(result).toBeTruthy();
+  });
+});
+

package/testdriver/acceptance-sdk/auto-cache-key-demo.test.mjs

@@ -0,0 +1,56 @@
+/**
+ * TestDriver SDK - Auto Cache Key Demo
+ * 
+ * This test demonstrates the auto-generated cache key feature.
+ * When no cacheKey is provided, TestDriver will automatically generate
+ * one based on the hash of this test file.
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+describe("Auto Cache Key Demo", () => {
+  it("should use auto-generated cache key based on file hash", async (context) => {
+    // NOTE: No cacheKey is provided here!
+    // TestDriver will automatically generate one from the hash of this file
+    const testdriver = TestDriver(context, { 
+      headless: true, 
+      newSandbox: true 
+      // cacheKey NOT specified - will be auto-generated
+    });
+    
+    // The cache key should be auto-generated
+    console.log('Auto-generated cache key:', testdriver.options.cacheKey);
+    expect(testdriver.options.cacheKey).toBeTruthy();
+    expect(testdriver.options.cacheKey).toMatch(/^[0-9a-f]{16}$/); // 16 hex chars
+    
+    await testdriver.provision.chrome({ url: 'http://testdriver-sandbox.vercel.app/login' });
+
+    // First find - will be cached with auto-generated key
+    const signInButton1 = await testdriver.find(
+      "Sign In, black button below the password field"
+    );
+    
+    // Second find - should hit cache because it's the same file (same cache key)
+    const signInButton2 = await testdriver.find(
+      "Sign In, black button below the password field"
+    );
+    
+    expect(signInButton1.found()).toBe(true);
+    expect(signInButton2.found()).toBe(true);
+  });
+
+  it("should use same auto-generated cache key for multiple tests in the same file", async (context) => {
+    // This test is in the same file, so it should get the same auto-generated cache key
+    const testdriver = TestDriver(context, { 
+      headless: true, 
+      newSandbox: true 
+    });
+    
+    console.log('Auto-generated cache key (test 2):', testdriver.options.cacheKey);
+    expect(testdriver.options.cacheKey).toBeTruthy();
+    
+    // If you modify this file, the hash (and therefore cache key) will change,
+    // invalidating the cache for this test file
+  });
+});

package/testdriver/acceptance-sdk/chrome-extension.test.mjs

@@ -0,0 +1,89 @@
+/**
+ * Chrome Extension Loading Demo
+ * Demonstrates how to load Chrome extensions using Chrome for Testing
+ * 
+ * This test shows how to launch Chrome with a specific extension loaded
+ * by using its Chrome Web Store extension ID.
+ */
+
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import TestDriver from "../../sdk.js";
+import {
+  runPostrun,
+  runPrerunChromeExtension
+} from "./setup/lifecycleHelpers.mjs";
+
+describe("Chrome Extension Loading", () => {
+  let client;
+  let dashcamUrl;
+
+  beforeAll(async () => {
+    // Initialize TestDriver client
+    client = await TestDriver.create({
+      apiKey: process.env.TD_API_KEY,
+      apiRoot: process.env.TD_API_ROOT,
+      os: "linux",
+      verbosity: 1,
+    });
+
+    // Run prerun with uBlock Origin extension loaded
+    // Extension ID: cjpalhdlnbpafiamejdnhcphjbkeiagm
+    await runPrerunChromeExtension(client, "cjpalhdlnbpafiamejdnhcphjbkeiagm");
+  });
+
+  afterAll(async () => {
+    if (client) {
+      dashcamUrl = await runPostrun(client);
+      await client.cleanup();
+    }
+  });
+
+  it("should load Chrome with extension and verify functionality", async () => {
+    // Focus Chrome browser
+    await client.focusApplication("Google Chrome");
+
+    // Verify the page loaded
+    const pageElement = await client.find("TestDriver.ai Sandbox");
+    expect(pageElement.found()).toBe(true);
+
+    // Test basic interaction to ensure Chrome is working with the extension
+    const signInButton = await client.find(
+      "Sign In, black button below the password field",
+    );
+    await signInButton.click();
+
+    // Verify error message appears
+    const result = await client.assert(
+      "an error shows that fields are required",
+    );
+    expect(result).toBeTruthy();
+
+    console.log("✅ Chrome extension loaded successfully!");
+    if (dashcamUrl) {
+      console.log("🎥 Dashcam URL:", dashcamUrl);
+    }
+  });
+
+  it("should demonstrate extension interaction", async () => {
+    // You can add specific tests here to interact with the extension
+    // For example, if testing uBlock Origin, you might:
+    // 1. Navigate to a page with ads
+    // 2. Verify ads are blocked
+    // 3. Access the extension's popup or settings
+    
+    await client.focusApplication("Google Chrome");
+    
+    // Example: Navigate to extension management page to verify it's loaded
+    await client.exec(
+      "sh",
+      `xdotool key --clearmodifiers ctrl+shift+e`,
+      5000,
+      true
+    );
+
+    // Wait a moment for the extensions page to potentially load
+    await new Promise((resolve) => setTimeout(resolve, 2000));
+
+    console.log("✅ Extension interaction test completed");
+  });
+});

package/testdriver/acceptance-sdk/drag-and-drop.test.mjs

@@ -0,0 +1,58 @@
+/**
+ * TestDriver SDK - Drag and Drop Test (Vitest)
+ * Converted from: testdriver/acceptance/drag-and-drop.yaml
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+const isLinux = (process.env.TD_OS || "linux") === "linux";
+
+describe("Drag and Drop Test", () => {
+  it.skipIf(isLinux)(
+    'should drag "New Text Document" to "Recycle Bin"',
+    async (context) => {
+      const testdriver = TestDriver(context, { headless: true });
+      await testdriver.provision.chrome({ url: 'http://testdriver-sandbox.vercel.app/login' });
+
+      //
+      // Show the desktop
+      await testdriver.pressKeys(["win", "d"]);
+
+      // Open the context menu
+      await testdriver.pressKeys(["shift", "f10"]);
+
+      // Hover over "New" in the context menu
+      const newOption = await testdriver.find(
+        "New, new option in the open context menu on the desktop",
+      );
+      await newOption.hover();
+
+      // Click "Text Document" in the context menu
+      const textDocOption = await testdriver.find(
+        "Text Document, text document option in the new submenu of the desktop context menu",
+      );
+      await textDocOption.click();
+
+      // Unfocus the "Text Document" text field
+      await testdriver.pressKeys(["esc"]);
+
+      // Drag the "New Text Document" icon to the "Recycle Bin"
+      const textDoc = await testdriver.find(
+        "New Text Document, new text document icon in the center of the desktop",
+      );
+      await textDoc.mouseDown();
+
+      const recycleBin = await testdriver.find(
+        "Recycle Bin, recycle bin icon in the top left corner of the desktop",
+      );
+      await recycleBin.mouseUp();
+
+      // Assert "New Text Document" icon is not on the Desktop
+      const result = await testdriver.assert(
+        'the "New Text Document" icon is not visible on the Desktop',
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});

package/testdriver/acceptance-sdk/element-not-found.test.mjs

@@ -0,0 +1,25 @@
+/**
+ * TestDriver SDK - Element Not Found Test
+ * Tests that finding a non-existent element returns properly without timing out
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+describe("Element Not Found Test", () => {
+  it("should handle non-existent element gracefully without timing out", async (context) => {
+    const testdriver = TestDriver(context, { headless: true });
+    await testdriver.provision.chrome({ url: 'http://testdriver-sandbox.vercel.app/login' });
+
+    //
+
+    // Try to find an element that definitely doesn't exist
+    const element = await testdriver.find(
+      "a purple unicorn dancing on the moon",
+    );
+
+    // Should return an element that is not found
+    expect(element.found()).toBe(false);
+    expect(element.coordinates).toBeNull();
+  }, 90000); // 90 second timeout for the test (should complete much faster)
+});

package/testdriver/acceptance-sdk/exec-js.test.mjs

@@ -0,0 +1,43 @@
+/**
+ * TestDriver SDK - Exec JS Test (Vitest)
+ * Converted from: testdriver/acceptance/exec-js.yaml
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+describe("Exec JavaScript Test", () => {
+  it("should fetch user data from API and enter email", async (context) => {
+    const testdriver = TestDriver(context, { headless: true });
+    await testdriver.provision.chrome({ url: 'http://testdriver-sandbox.vercel.app/login' });
+
+    //
+    // Execute JavaScript to fetch user data
+    const userEmail = await testdriver.exec(
+      "js",
+      `
+      const response = await fetch('https://jsonplaceholder.typicode.com/users');
+      const user = await response.json();
+      console.log('user', user[0]);
+      result = user[0].email;
+    `,
+      10000,
+    );
+
+    expect(userEmail).toBeTruthy();
+    expect(userEmail).toContain("@");
+
+    // Enter email in username field
+    const usernameField = await testdriver.find(
+      "Username, input field for username",
+    );
+    await usernameField.click();
+    await testdriver.type(userEmail);
+
+    // Assert email is in the field
+    const result = await testdriver.assert(
+      'the username field contains "Sincere@april.biz" which is a valid email address',
+    );
+    expect(result).toBeTruthy();
+  });
+});

package/testdriver/acceptance-sdk/exec-output.test.mjs

@@ -0,0 +1,59 @@
+/**
+ * TestDriver SDK - Exec Output Test (Vitest)
+ * Converted from: testdriver/acceptance/exec-output.yaml
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+describe("Exec Output Test", () => {
+  it.skipIf(process.env.TD_OS === "linux")(
+    "should set date using PowerShell and navigate to calendar",
+    async (context) => {
+      const testdriver = TestDriver(context, { headless: true });
+      await testdriver.provision.chrome({ url: 'http://testdriver-sandbox.vercel.app/login' });
+
+      //
+      // Generate date in query string format
+      const queryString = await testdriver.exec(
+        "pwsh",
+        `
+$date = (Get-Date).AddMonths(1)
+Write-Output $date.ToString("yyyy-MM-dd")
+    `,
+        10000,
+      );
+
+      // Assert that the date is valid
+      const dateValidResult = await testdriver.assert(
+        `${queryString} is a valid date`,
+      );
+      expect(dateValidResult).toBeTruthy();
+
+      // Generate date in display format
+      const expectedDate = await testdriver.exec(
+        "pwsh",
+        `
+$date = (Get-Date).AddMonths(1)
+Write-Output $date.ToString("ddd MMM d yyyy")
+    `,
+        10000,
+      );
+
+      // Navigate to calendar with date parameter
+      await testdriver.focusApplication("Google Chrome");
+      await testdriver.pressKeys(["ctrl", "l"]);
+      await testdriver.type(
+        `https://teamup.com/ks48cf2135e7e080bc?view=d&date=${queryString}`,
+      );
+      await testdriver.pressKeys(["enter"]);
+
+      // Assert that the expected date shows
+      await testdriver.focusApplication("Google Chrome");
+      const result = await testdriver.assert(
+        `the text ${expectedDate} is visible on screen`,
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});

package/testdriver/acceptance-sdk/exec-pwsh.test.mjs

@@ -0,0 +1,57 @@
+/**
+ * TestDriver SDK - Exec Shell Test (Vitest)
+ * Converted from: testdriver/acceptance/exec-shell.yaml
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+describe("Exec PowerShell Test", () => {
+  it.skipIf(process.env.TD_OS === "linux")(
+    "should generate random email using PowerShell and enter it",
+    async (context) => {
+      const testdriver = TestDriver(context, { headless: true });
+      await testdriver.provision.chrome({ url: 'http://testdriver-sandbox.vercel.app/login' });
+
+      //
+      // Generate random email using PowerShell
+      const randomEmail = await testdriver.exec({
+        language: "pwsh",
+        code: `
+# Random email generator in PowerShell
+
+# Arrays of possible names and domains
+$firstNames = @("john", "jane", "alex", "chris", "sara", "mike", "lisa", "david", "emma", "ryan")
+$lastNames = @("smith", "johnson", "williams", "brown", "jones", "garcia", "miller", "davis", "martin", "lee")
+$domains = @("example.com", "testmail.com", "mailinator.com", "demo.org", "company.net")
+
+# Random selection
+$first = Get-Random -InputObject $firstNames
+$last = Get-Random -InputObject $lastNames
+$domain = Get-Random -InputObject $domains
+$number = Get-Random -Minimum 1 -Maximum 1000
+
+# Generate the email
+$email = "$first.$last$number@$domain".ToLower()
+
+# Output
+Write-Output "$email"
+    `,
+        timeout: 10000,
+      });
+
+      // Enter the email in username field
+      const usernameField = await testdriver.find(
+        "Username, input field for username",
+      );
+      await usernameField.click();
+      await testdriver.type(randomEmail);
+
+      // Assert that the username field shows a valid email address
+      const result = await testdriver.assert(
+        `the username field contains ${randomEmail} which is a valid email address`,
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});

package/testdriver/acceptance-sdk/focus-window.test.mjs

@@ -0,0 +1,36 @@
+/**
+ * TestDriver SDK - Focus Window Test (Vitest)
+ * Converted from: testdriver/acceptance/focus-window.yaml
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../../src/vitest/hooks.mjs";
+
+describe("Focus Window Test", () => {
+  it.skipIf(process.env.TD_OS === "linux")(
+    "should click Microsoft Edge icon and focus Google Chrome",
+    async (context) => {
+      const testdriver = TestDriver(context, { headless: true });
+      await testdriver.provision.chrome({ url: 'http://testdriver-sandbox.vercel.app/login' });
+
+      //
+      // Show desktop
+      await testdriver.pressKeys(["winleft", "d"]);
+
+      // Click on the Microsoft Edge icon
+      const edgeIcon = await testdriver.find(
+        "a blue and green swirl icon on the taskbar representing Microsoft Edge",
+      );
+      await edgeIcon.click();
+
+      // Focus Google Chrome
+      await testdriver.focusApplication("Google Chrome");
+
+      // Assert Chrome is focused (implicit through successful focus)
+      const result = await testdriver.assert(
+        "Google Chrome is the focused application",
+      );
+      expect(result).toBeTruthy();
+    },
+  );
+});