testdriverai 7.9.0-test.4 → 7.9.0-test.40

package/ai/skills/testdriver-test-results-json/SKILL.md

@@ -0,0 +1,257 @@
+---
+name: testdriver:test-results-json
+description: Per-test JSON result files with metadata, versions, and infrastructure details
+---
+<!-- Generated from test-results-json.mdx. DO NOT EDIT. -->
+
+## Overview
+
+TestDriver automatically writes a JSON result file for each test case after it finishes. These files contain comprehensive metadata about the test run, including SDK and runner versions, infrastructure details, interaction statistics, and links to recordings.
+
+Result files are written to:
+
+```
+.testdriver/results/<testFile>/<testName>.json
+```
+
+For example, a test file `tests/login.test.mjs` with a test named `"should log in"` produces:
+
+```
+.testdriver/results/tests/login.test.mjs/should_log_in.json
+```
+
+<Note>
+  Test names are sanitized for filesystem use — special characters are replaced with underscores and names are truncated to 200 characters.
+</Note>
+
+## Enabling
+
+No configuration is required. The JSON files are written automatically by the TestDriver Vitest reporter plugin whenever tests run.
+
+## JSON Schema
+
+Each result file is organized into logical groups:
+
+### `versions`
+
+| Field | Type | Description |
+|---|---|---|
+| `versions.sdk` | `string \| null` | TestDriver SDK version (e.g. `"7.8.0"`) |
+| `versions.vitest` | `string \| null` | Vitest version used to run the test |
+| `versions.api` | `string \| null` | TestDriver API server version |
+| `versions.runnerBefore` | `string \| null` | Runner version at sandbox start |
+| `versions.runnerAfter` | `string \| null` | Runner version after auto-update |
+| `versions.runnerWasUpdated` | `boolean` | Whether the runner was auto-updated during provisioning |
+
+### `test`
+
+| Field | Type | Description |
+|---|---|---|
+| `test.file` | `string \| null` | Relative path to the test file |
+| `test.name` | `string \| null` | Name of the test case |
+| `test.suite` | `string \| null` | Name of the parent `describe` block |
+| `test.passed` | `boolean` | Whether the test passed |
+| `test.caseId` | `string \| null` | Database ID for this test case |
+| `test.runId` | `string \| null` | Database ID for the overall test run |
+| `test.error` | `string \| null` | Error message if the test failed |
+| `test.errorStack` | `string \| null` | Error stack trace if the test failed |
+
+### `urls`
+
+| Field | Type | Description |
+|---|---|---|
+| `urls.api` | `string \| null` | API root URL used for this test |
+| `urls.console` | `string \| null` | TestDriver console base URL |
+| `urls.vnc` | `string \| null` | VNC URL for the sandbox |
+| `urls.testRun` | `string \| null` | Direct link to this test case in the console |
+
+### `replay`
+
+The `replay` object contains the recording replay URL and derived embed links. The `gifUrl` and `embedUrl` are generated automatically from the replay URL.
+
+| Field | Type | Description |
+|---|---|---|
+| `replay.url` | `string \| null` | Recording replay URL |
+| `replay.gifUrl` | `string \| null` | Animated GIF thumbnail of the recording |
+| `replay.embedUrl` | `string \| null` | Embeddable replay URL (appends `&embed=true`) |
+| `replay.markdown` | `string \| null` | Ready-to-use Markdown embed with GIF linking to the replay |
+
+The `replay.markdown` field produces a clickable GIF badge you can paste directly into PR comments, README files, or issue descriptions:
+
+```markdown
+[![Test Recording](https://api.testdriver.ai/replay/abc123/gif?shareKey=xyz)](https://console.testdriver.ai/replay/abc123?share=xyz)
+```
+
+### `date`
+
+| Field | Type | Description |
+|---|---|---|
+| `date` | `string` | ISO 8601 timestamp when the test finished |
+
+### `team`
+
+| Field | Type | Description |
+|---|---|---|
+| `team.id` | `string \| null` | Team ID from the sandbox |
+| `team.sessionId` | `string \| null` | SDK session ID |
+
+### `infrastructure`
+
+| Field | Type | Description |
+|---|---|---|
+| `infrastructure.sandboxId` | `string \| null` | Sandbox instance ID |
+| `infrastructure.instanceId` | `string \| null` | Instance ID |
+| `infrastructure.os` | `string \| null` | Operating system of the sandbox (`"linux"` or `"windows"`) |
+| `infrastructure.amiId` | `string \| null` | AWS AMI ID used for provisioning |
+| `infrastructure.e2bTemplateId` | `string \| null` | E2B template ID used for provisioning |
+| `infrastructure.imageVersion` | `string \| null` | Sandbox image version |
+
+### `realtime`
+
+| Field | Type | Description |
+|---|---|---|
+| `realtime.channel` | `string \| null` | Ably channel name used for communication |
+| `realtime.messageCount` | `number` | Number of messages published to the realtime channel |
+
+### `interactions`
+
+| Field | Type | Description |
+|---|---|---|
+| `interactions.total` | `number` | Total number of interactions recorded |
+| `interactions.cached` | `number` | Number of interactions served from cache |
+| `interactions.byType` | `object` | Breakdown of interactions by type (e.g. `find`, `click`, `assert`) |
+
+## Example Output
+
+```json
+{
+  "sdkVersion": "7.8.0",
+  "vitestVersion": "4.0.0",
+  "apiVersion": "1.45.0",
+  "runnerVersionBefore": "2.1.0",
+  "runnerVersionAfter": "2.1.1",
+  "wasUpdated": true,
+  "apiUrl": "https://api.testdriver.ai",
+  "consoleUrl": "https://console.testdriver.ai",
+  "testRunLink": "https://console.testdriver.ai/runs/abc123/def456",
+  "dashcamUrl": "https://app.dashcam.io/replay/abc123",
+  "vncUrl": "wss://sandbox-123.testdriver.ai/vnc",
+  "date": "2025-01-15T14:30:00.000Z",
+  "team": {
+    "id": "team_abc123",
+    "sessionId": "sess_xyz789"
+  },
+  "infrastructure": {
+    "sandboxId": "sandbox-123",
+    "instanceId": "i-abc123",
+    "os": "linux",
+    "amiId": "ami-0abc123",
+    "e2bTemplateId": null,
+    "imageVersion": "v2.1.0"
+  },
+  "realtime": {
+    "channel": "sandbox:sandbox-123",
+    "messageCount": 42
+  },
+  "interactions": {
+    "total": 15,
+    "cached": 3,
+    "byType": {
+      "find": 8,
+      "click": 5,
+      "assert": 2
+    }
+  }
+}
+```
+
+## Using Result Files in CI
+
+Result files are useful for extracting test metadata in CI pipelines without parsing log output.
+
+### GitHub Actions Example
+
+Use `fromJSON` to parse a result file into a GitHub Actions expression you can reference in subsequent steps:
+
+```yaml
+- name: Run tests
+  run: npx vitest run tests/login.test.mjs
+
+- name: Parse result
+  id: result
+  run: |
+    # Read the first JSON result file
+    FILE=$(find .testdriver/results -name '*.json' | head -n 1)
+    echo "json=$(cat "$FILE")" >> "$GITHUB_OUTPUT"
+
+- name: Comment on PR
+  if: fromJSON(steps.result.outputs.json).test.passed == false
+  uses: actions/github-script@v7
+  with:
+    script: |
+      const result = ${{ steps.result.outputs.json }};
+      await github.rest.issues.createComment({
+        owner: context.repo.owner,
+        repo: context.repo.repo,
+        issue_number: context.issue.number,
+        body: [
+          `❌ **${result.test.name}** failed`,
+          ``,
+          `Error: ${result.test.error}`,
+          ``,
+          result.replay.markdown,
+          ``,
+          `[View full recording](${result.urls.testRun})`
+        ].join('\n')
+      });
+```
+
+You can also load all results into a matrix or iterate over them:
+
+```yaml
+- name: Run tests
+  run: npx vitest run tests/*.test.mjs
+
+- name: Collect results
+  id: results
+  run: |
+    # Merge all result files into a JSON array
+    echo "json=$(find .testdriver/results -name '*.json' -exec cat {} + | jq -s '.')" >> "$GITHUB_OUTPUT"
+
+- name: Summary
+  run: |
+    echo '## Test Results' >> $GITHUB_STEP_SUMMARY
+    RESULTS='${{ steps.results.outputs.json }}'
+    echo "$RESULTS" | jq -r '.[] | "| \(.test.name) | \(if .test.passed then "✅" else "❌" end) | \(.urls.testRun) |"' >> $GITHUB_STEP_SUMMARY
+```
+
+### Reading Results Programmatically
+
+```javascript
+import fs from "fs";
+import path from "path";
+
+const resultsDir = ".testdriver/results";
+
+function readResults(dir) {
+  const results = [];
+  for (const testDir of fs.readdirSync(dir, { recursive: true })) {
+    const fullPath = path.join(dir, testDir);
+    if (fullPath.endsWith(".json") && fs.statSync(fullPath).isFile()) {
+      results.push(JSON.parse(fs.readFileSync(fullPath, "utf-8")));
+    }
+  }
+  return results;
+}
+
+const results = readResults(resultsDir);
+const passed = results.filter(r => r.test.passed);
+const failed = results.filter(r => !r.test.passed);
+
+console.log(`${passed.length} passed, ${failed.length} failed`);
+for (const r of failed) {
+  console.log(`  FAIL: ${r.test.name} — ${r.test.error}`);
+  console.log(`  Recording: ${r.urls.testRun}`);
+  console.log(`  Embed: ${r.replay.markdown}`);
+}
+```

package/docs/_scripts/generate-examples.js

@@ -297,8 +297,9 @@ function updateExistingMDX(existingContent, filename, testcaseId) {
   const replayUrl = generateReplayUrl(testcaseId);
   
   // Pattern to match the marker followed by the iframe tag
+  const escapedFilename = filename.replace(/\./g, '\\.');
   const pattern = new RegExp(
-    `(\\{/\\* ${filename.replace('.', '\\.')} output \\*/\\}\\s*)<iframe[^>]*src="[^"]*"([^]*)/>`,
+    `(\\{/\\* ${escapedFilename} output \\*/\\}\\s*)<iframe[^>]*src="[^"]*"([^]*?)/>`,
     's'
   );
   
@@ -312,6 +313,25 @@ function updateExistingMDX(existingContent, filename, testcaseId) {
   return updated;
 }
 
+// Update the source code block in an existing MDX file with fresh content from the test file
+function updateSourceCodeBlock(existingContent, testMeta) {
+  const escapedFilename = testMeta.filename.replace(/\./g, '\\.');
+  const codeBlockPattern = new RegExp(
+    '(```javascript\\s+title="' + escapedFilename + '"[^\\n]*\\n)[\\s\\S]*?(\\n```)',
+    ''
+  );
+
+  const match = existingContent.match(codeBlockPattern);
+  if (!match) {
+    return null;
+  }
+
+  return existingContent.replace(
+    codeBlockPattern,
+    `$1${testMeta.content.trim()}$2`
+  );
+}
+
 // Generate MDX content
 function generateMDX(testMeta, manifest, description) {
   const slug = generateSlug(testMeta.filename);
@@ -399,13 +419,29 @@ function updateDocsNavigation(docsJson, examplePages, options) {
     return false;
   }
 
-  // Find or create Examples group
-  let examplesGroup = v7Version.groups.find((g) => 
-    g.group === "Examples" || 
-    (typeof g === "object" && g.group === "Examples")
+  // Find Examples group - it may be nested inside Overview's pages
+  const examplesPages = examplePages.map((slug) => `/v7/examples/${slug}`);
+  let examplesGroup = null;
+
+  // Search top-level groups first
+  examplesGroup = v7Version.groups.find((g) =>
+    typeof g === "object" && g.group === "Examples"
   );
 
-  const examplesPages = examplePages.map((slug) => `/v7/examples/${slug}`);
+  // If not found at top level, search inside each group's pages (nested groups)
+  if (!examplesGroup) {
+    for (const group of v7Version.groups) {
+      if (group.pages) {
+        const nested = group.pages.find((p) =>
+          typeof p === "object" && p.group === "Examples"
+        );
+        if (nested) {
+          examplesGroup = nested;
+          break;
+        }
+      }
+    }
+  }
 
   if (examplesGroup) {
     // Update existing group
@@ -414,20 +450,22 @@ function updateDocsNavigation(docsJson, examplePages, options) {
       console.log("🔄 Updated existing Examples group in navigation");
     }
   } else {
-    // Create new group after Overview
-    const overviewIndex = v7Version.groups.findIndex((g) => g.group === "Overview");
+    // Create new group nested inside Overview
+    const overviewGroup = v7Version.groups.find((g) => g.group === "Overview");
     const newGroup = {
       group: "Examples",
       icon: "code",
       pages: examplesPages,
     };
-    
-    if (overviewIndex !== -1) {
-      v7Version.groups.splice(overviewIndex + 1, 0, newGroup);
+
+    if (overviewGroup && overviewGroup.pages) {
+      // Insert after the second page in Overview (after what-is-testdriver)
+      const insertIdx = Math.min(2, overviewGroup.pages.length);
+      overviewGroup.pages.splice(insertIdx, 0, newGroup);
     } else {
       v7Version.groups.push(newGroup);
     }
-    
+
     if (options.verbose) {
       console.log("➕ Added new Examples group to navigation");
     }
@@ -439,7 +477,7 @@ function updateDocsNavigation(docsJson, examplePages, options) {
 // Show help
 function showHelp() {
   console.log(`
-Update Example Docs Iframe URLs
+Update Example Docs from Test Files
 
 Usage:
   node generate-examples.js [options]
@@ -453,12 +491,16 @@ Environment Variables:
   TD_API_ROOT       API root URL (default: https://api.testdriver.ai)
 
 Description:
-  Reads existing MDX files in docs/v7/examples/ and updates the iframe
-  src URLs based on the examples-manifest.json.
+  Reads example test files and updates/generates MDX documentation pages.
+  
+  For existing MDX files:
+    - Updates the source code block from the test file
+    - Updates the iframe src URL from examples-manifest.json
+  
+  For new test files without an MDX page:
+    - Generates a new MDX documentation page
   
-  Files must contain a marker comment like: {/* filename.test.mjs output */}
-  The iframe following the marker will have its src updated to the
-  API replay endpoint.
+  Also updates the docs.json navigation with the current example pages.
 `);
 }
 
@@ -471,7 +513,7 @@ async function main() {
     process.exit(0);
   }
 
-  console.log("🚀 Updating example documentation iframes...\n");
+  console.log("🚀 Updating example documentation...\n");
 
   if (options.dryRun) {
     console.log("📋 DRY RUN - no files will be written\n");
@@ -480,67 +522,92 @@ async function main() {
   // Load manifest
   const manifest = loadManifest();
 
-  // Get existing MDX files in output directory
-  const existingFiles = fs.existsSync(OUTPUT_DIR) 
-    ? fs.readdirSync(OUTPUT_DIR).filter((f) => f.endsWith(".mdx"))
-    : [];
+  // Get all test files
+  const testFiles = getExampleFiles();
+  console.log(`📂 Found ${testFiles.length} example test files\n`);
 
-  console.log(`📂 Found ${existingFiles.length} existing MDX files\n`);
+  // Ensure output dir exists
+  if (!fs.existsSync(OUTPUT_DIR)) {
+    fs.mkdirSync(OUTPUT_DIR, { recursive: true });
+  }
 
   let updated = 0;
+  let created = 0;
   let skipped = 0;
   let errors = 0;
+  const exampleSlugs = [];
+
+  for (const file of testFiles) {
+    const filePath = path.join(EXAMPLES_DIR, file);
+    const testMeta = parseTestFile(filePath);
+    const slug = generateSlug(testMeta.filename);
+    exampleSlugs.push(slug);
 
-  for (const mdxFile of existingFiles) {
-    const outputPath = path.join(OUTPUT_DIR, mdxFile);
+    const outputPath = path.join(OUTPUT_DIR, `${slug}.mdx`);
 
     try {
-      const existingContent = fs.readFileSync(outputPath, 'utf-8');
-      
-      // Find the marker in the file to get the test filename
-      const markerMatch = existingContent.match(/\{\/\* ([^*]+\.test\.mjs) output \*\/\}/);
-      
-      if (!markerMatch) {
-        skipped++;
-        if (options.verbose) {
-          console.log(`⏭️  ${mdxFile} (no marker)`);
+      if (fs.existsSync(outputPath)) {
+        // Update existing MDX file
+        let content = fs.readFileSync(outputPath, 'utf-8');
+        let changed = false;
+
+        // Update source code block from the test file
+        const sourceUpdated = updateSourceCodeBlock(content, testMeta);
+        if (sourceUpdated && sourceUpdated !== content) {
+          content = sourceUpdated;
+          changed = true;
         }
-        continue;
-      }
-      
-      const testFilename = markerMatch[1];
-      const manifestEntry = manifest.examples[testFilename];
-      const testcaseId = manifestEntry?.url ? extractTestcaseId(manifestEntry.url) : null;
-      
-      if (!testcaseId) {
-        skipped++;
-        if (options.verbose) {
-          console.log(`⏭️  ${mdxFile} (no URL in manifest for ${testFilename})`);
+
+        // Update iframe URL if manifest entry exists
+        const manifestEntry = manifest.examples[testMeta.filename];
+        const testcaseId = manifestEntry?.url ? extractTestcaseId(manifestEntry.url) : null;
+        if (testcaseId) {
+          const iframeUpdated = updateExistingMDX(content, testMeta.filename, testcaseId);
+          if (iframeUpdated && iframeUpdated !== content) {
+            content = iframeUpdated;
+            changed = true;
+          }
         }
-        continue;
-      }
-      
-      const updatedContent = updateExistingMDX(existingContent, testFilename, testcaseId);
-      
-      if (updatedContent) {
-        if (!options.dryRun) {
-          fs.writeFileSync(outputPath, updatedContent, 'utf-8');
+
+        if (changed) {
+          if (!options.dryRun) {
+            fs.writeFileSync(outputPath, content, 'utf-8');
+          }
+          updated++;
+          console.log(`🔄 ${slug}.mdx (updated)`);
+        } else {
+          skipped++;
+          if (options.verbose) {
+            console.log(`⏭️  ${slug}.mdx (unchanged)`);
+          }
         }
-        updated++;
-        console.log(`🔄 ${mdxFile} (updated iframe)`);
       } else {
-        skipped++;
-        if (options.verbose) {
-          console.log(`⏭️  ${mdxFile} (unchanged)`);
+        // Generate new MDX file
+        const description = generateFallbackDescription(testMeta);
+        const mdx = generateMDX(testMeta, manifest, description);
+        if (!options.dryRun) {
+          fs.writeFileSync(outputPath, mdx, 'utf-8');
         }
+        created++;
+        console.log(`✨ ${slug}.mdx (created)`);
       }
     } catch (error) {
-      console.error(`❌ ${mdxFile}: ${error.message}`);
+      console.error(`❌ ${file}: ${error.message}`);
       errors++;
     }
   }
 
+  // Update docs.json navigation
+  if (!options.dryRun) {
+    const docsJson = loadDocsJson();
+    if (updateDocsNavigation(docsJson, exampleSlugs, options)) {
+      saveDocsJson(docsJson);
+      console.log("\n📝 Updated docs.json navigation");
+    }
+  }
+
   console.log(`\n✨ Complete!`);
+  console.log(`   Created: ${created} new docs`);
   console.log(`   Updated: ${updated} docs`);
   console.log(`   Skipped: ${skipped} unchanged`);
   if (errors > 0) {

package/docs/docs.json

@@ -27,28 +27,29 @@
                 "pages": [
                   "/v7/examples/ai",
                   "/v7/examples/assert",
-                  "/v7/examples/captcha-api",
                   "/v7/examples/chrome-extension",
-                  "/v7/examples/drag-and-drop",
                   "/v7/examples/element-not-found",
                   "/v7/examples/exec-output",
                   "/v7/examples/exec-pwsh",
+                  "/v7/examples/findall-coffee-icons",
                   "/v7/examples/focus-window",
+                  "/v7/examples/formatted-logging",
                   "/v7/examples/hover-image",
+                  "/v7/examples/hover-text-with-description",
                   "/v7/examples/hover-text",
                   "/v7/examples/installer",
                   "/v7/examples/launch-vscode-linux",
                   "/v7/examples/match-image",
+                  "/v7/examples/parse",
                   "/v7/examples/press-keys",
+                  "/v7/examples/prompt",
                   "/v7/examples/scroll-keyboard",
                   "/v7/examples/scroll-until-image",
-                  "/v7/examples/scroll-until-text",
                   "/v7/examples/scroll",
                   "/v7/examples/type",
                   "/v7/examples/windows-installer"
                 ]
               },
-
               {
                 "group": "Deployment",
                 "icon": "server",
@@ -60,7 +61,6 @@
               "/changelog"
             ]
           },
-
           {
             "group": "GitHub Copilot",
             "pages": [
@@ -106,34 +106,33 @@
               }
             ]
           },
+          {
+            "group": "Actions",
+            "pages": [
+              "/v7/ai",
+              "/v7/assert",
+              "/v7/captcha",
+              "/v7/click",
+              "/v7/double-click",
+              "/v7/exec",
+              "/v7/find",
+              "/v7/focus-application",
+              "/v7/hover",
+              "/v7/mouse-down",
+              "/v7/mouse-up",
+              "/v7/parse",
+              "/v7/press-keys",
+              "/v7/right-click",
+              "/v7/screenshot",
+              "/v7/scroll",
+              "/v7/type"
+            ]
+          },
           {
             "group": "SDK Reference",
             "pages": [
               "/v7/client",
               "/v7/elements",
-              {
-                "group": "Actions",
-                "icon": "bolt",
-                "pages": [
-                  "/v7/ai",
-                  "/v7/assert",
-                  "/v7/captcha",
-                  "/v7/click",
-                  "/v7/double-click",
-                  "/v7/exec",
-                  "/v7/find",
-                  "/v7/focus-application",
-                  "/v7/hover",
-                  "/v7/mouse-down",
-                  "/v7/mouse-up",
-                  "/v7/parse",
-                  "/v7/press-keys",
-                  "/v7/right-click",
-                  "/v7/screenshot",
-                  "/v7/scroll",
-                  "/v7/type"
-                ]
-              },
               "/v7/dashcam"
             ]
           }

package/docs/v7/examples/ai.mdx

@@ -12,7 +12,7 @@ Watch this test execute in a real sandbox environment:
 
 {/* ai.test.mjs output */}
 <iframe 
-  src="https://api-test.testdriver.ai/api/v1/testdriver/testcase/69c5d04ac2e6e94933886778/replay" 
+  src="https://api.testdriver.ai/api/v1/testdriver/testcase/69c5d04ac2e6e94933886778/replay" 
   width="100%" 
   height="390" 
   style={{ border: "1px solid #333", borderRadius: "8px" }}
@@ -23,7 +23,8 @@ Watch this test execute in a real sandbox environment:
 
 ```javascript title="ai.test.mjs" {17}
 /**
- * TestDriver SDK - AI Test
+ * TestDriver SDK - AI Test (Vitest)
+ * Tests the AI exploratory loop (ai) functionality
  */
 
 import { describe, expect, it } from "vitest";
@@ -31,7 +32,7 @@ import { TestDriver } from "testdriverai/vitest/hooks";
 
 describe("AI Test", () => {
   it("should use ai to search for testdriver on Google", async (context) => {
-    const testdriver = TestDriver(context, { ip: context.ip || process.env.TD_IP });
+    const testdriver = TestDriver(context, { ip: context.ip || process.env.TD_IP});
     
     // provision.chrome() automatically calls ready() and starts dashcam
     await testdriver.provision.chrome({