testdriverai 7.3.1 → 7.3.3

package/docs/GITHUB_COMMENTS_ANNOUNCEMENT.md

@@ -0,0 +1,167 @@
+# 🎉 GitHub Comments Feature
+
+## What's New
+
+TestDriver now automatically posts beautiful, detailed comments to your GitHub pull requests and commits with:
+
+- ✅ **Test Results** - Pass/fail statistics at a glance
+- 🎥 **Dashcam GIF Replays** - Embedded animated previews of each test
+- 📊 **Detailed Statistics** - Duration, platform, branch, commit info
+- ❌ **Exception Details** - Full error messages and stack traces for failures
+- 🔄 **Smart Updates** - Updates existing comments instead of creating duplicates
+
+## Quick Start
+
+### 1. Add to GitHub Actions
+
+```yaml
+- name: Run TestDriver tests
+  env:
+    TD_API_KEY: ${{ secrets.TD_API_KEY }}
+    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    GITHUB_PR_NUMBER: ${{ github.event.pull_request.number }}
+  run: npm run test:sdk
+```
+
+### 2. That's it!
+
+TestDriver will automatically detect the CI environment and post comments when:
+- `GITHUB_TOKEN` is present
+- `GITHUB_PR_NUMBER` (for PR comments) or `GITHUB_SHA` (for commit comments) is set
+
+## Example Comment
+
+![GitHub Comment Preview](https://placeholder.com/github-comment-preview.png)
+
+Your team will see:
+- **Status badges** with pass/fail counts
+- **Embedded GIF replays** directly in the comment
+- **Clickable links** to full test runs and individual replays
+- **Collapsible error details** with full stack traces
+
+## Features
+
+### 🎥 Dashcam Replays
+
+Every test automatically records a dashcam replay. These appear in the GitHub comment as:
+1. **GIF preview** - Shows the test execution as an animated image
+2. **Link to full replay** - Opens the complete replay in TestDriver console
+3. **Test-specific context** - Each test has its own replay section
+
+### 📊 Test Statistics
+
+The comment shows:
+- Total test count
+- Pass/fail/skip breakdown
+- Total execution time
+- Platform (Linux/Windows/Mac)
+- Git branch and commit
+
+### ❌ Exception Handling
+
+Failed tests include:
+- Error message prominently displayed
+- Collapsible stack trace
+- Direct link to the failing test's replay
+- File and line number information
+
+### 🔄 Comment Updates
+
+TestDriver intelligently:
+- **Updates existing comments** on subsequent pushes to the same PR
+- **Creates new comments** only when needed
+- **Identifies its own comments** using a signature
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `TD_API_KEY` | TestDriver API key | ✅ Yes |
+| `GITHUB_TOKEN` | GitHub token | ✅ Yes |
+| `GITHUB_PR_NUMBER` | PR number for PR comments | 📝 Recommended |
+| `GITHUB_SHA` | Commit SHA (fallback) | ⚙️ Auto-detected |
+| `TESTDRIVER_SKIP_GITHUB_COMMENT` | Set to 'true' to disable | ❌ No |
+
+### Disabling Comments
+
+To disable GitHub comments for a specific run:
+
+```bash
+TESTDRIVER_SKIP_GITHUB_COMMENT=true npm run test
+```
+
+## Documentation
+
+For complete documentation, see:
+- **[GitHub Comments Guide](./GITHUB_COMMENTS.md)** - Full setup and configuration
+- **[Example Workflow](./../examples/github-actions.yml)** - Copy-paste GitHub Actions workflow
+- **[Example Test](./../examples/github-comment-demo.test.mjs)** - Demo test file
+
+## Troubleshooting
+
+### Comment not appearing?
+
+1. **Check environment variables:**
+   ```bash
+   echo "Token: ${GITHUB_TOKEN:0:10}..."
+   echo "PR: $GITHUB_PR_NUMBER"
+   ```
+
+2. **Verify permissions:**
+   - GitHub Actions needs `pull-requests: write` or `contents: write`
+
+3. **Check logs:**
+   - Look for "Posting GitHub comment..." in test output
+   - Check for "GitHub token not found" warnings
+
+### Replays not embedded?
+
+- Ensure tests are using `await testdriver.provision.*()` 
+- Check that dashcam stops successfully (look for "🎥 Dashcam URL")
+- Verify replay URLs in test output
+
+## Examples
+
+### Basic Test with Comment
+
+```javascript
+it("should login successfully", async (context) => {
+  const testdriver = TestDriver(context, { headless: true });
+  
+  await testdriver.provision.chrome({
+    url: 'https://your-app.com/login',
+  });
+
+  // Test steps...
+  const result = await testdriver.assert("I'm logged in");
+  expect(result).toBeTruthy();
+  
+  // Dashcam automatically recorded and will appear in GitHub comment!
+});
+```
+
+### Local Testing
+
+Run locally without posting comments:
+
+```bash
+npm run test:sdk
+```
+
+Run locally WITH comment posting (requires token):
+
+```bash
+GITHUB_TOKEN=ghp_xxx GITHUB_PR_NUMBER=123 npm run test:sdk
+```
+
+## Support
+
+- **Documentation:** [testdriver.ai/docs](https://testdriver.ai/docs)
+- **Issues:** [GitHub Issues](https://github.com/testdriverai/testdriverai/issues)
+- **Discord:** [Join our community](https://discord.gg/testdriver)
+
+---
+
+Made with ❤️ by the TestDriver team

package/docs/QUICK-START-GITHUB-COMMENTS.md

@@ -0,0 +1,84 @@
+# Quick Start: GitHub Comments
+
+## For GitHub Actions (Easiest!)
+
+Add this to your `.github/workflows/test.yml`:
+
+```yaml
+name: Tests
+
+on: pull_request
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write  # ← This is all you need!
+      
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Setup Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          
+      - run: npm ci
+      
+      - name: Run TestDriver tests
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # Auto-provided
+        run: vitest run
+```
+
+That's it! TestDriver will automatically:
+- ✅ Detect the PR number
+- ✅ Detect the repository
+- ✅ Post beautiful comments with test results
+- 🎥 Embed dashcam GIF replays
+- 📊 Show pass/fail statistics
+- 🔴 Display exception details
+
+## For Local Development
+
+1. **Create a GitHub Personal Access Token**
+   - Go to https://github.com/settings/tokens
+   - Click "Generate new token (classic)"
+   - Select scope: `repo` or `public_repo`
+   - Copy the token
+
+2. **Set environment variables**
+   ```bash
+   export GITHUB_TOKEN=ghp_your_token_here
+   export GITHUB_PR_NUMBER=123
+   vitest run
+   ```
+
+## What Gets Posted
+
+```markdown
+# 🟢 TestDriver Test Results
+
+**Status:** ✅ PASSED
+**Duration:** 12.3s
+
+## 📝 Test Results
+| Status | Test | Duration | Replay |
+|--------|------|----------|--------|
+| ✅ | Login test | 4.5s | [🎥 View](link) |
+
+## 🎥 Dashcam Replays
+[Animated GIF embedded here]
+```
+
+## Disable Comments
+
+```bash
+TESTDRIVER_SKIP_GITHUB_COMMENT=true vitest run
+```
+
+## Full Documentation
+
+See [github-integration-setup.md](./github-integration-setup.md) for complete details.

package/docs/TEST-GITHUB-COMMENTS.md

@@ -0,0 +1,129 @@
+# Testing GitHub Comments Feature
+
+## Quick Test Instructions
+
+1. **Ensure TD_API_KEY is set**
+   - Go to: https://github.com/testdriverai/testdriverai/settings/secrets/actions
+   - Add secret named `TD_API_KEY` with your API key from https://console.testdriver.ai/team
+   - If already set, you're good to go!
+
+2. **Create a test branch and PR**
+   ```bash
+   cd /Users/ianjennings/Development/testdriverai
+   git checkout -b test-github-comments
+   
+   # Make a small change to trigger the workflow
+   echo "# Test GitHub Comments" >> test-comment.md
+   git add test-comment.md
+   git commit -m "test: trigger GitHub comment workflow"
+   git push origin test-github-comments
+   ```
+
+3. **Create Pull Request**
+   ```bash
+   gh pr create --title "Test: GitHub Comments Feature" \
+                --body "Testing automatic GitHub comments with test results and dashcam replays"
+   ```
+   
+   Or create via web: https://github.com/testdriverai/testdriverai/compare
+
+4. **Watch the Magic Happen! ✨**
+   - The workflow will automatically run: `.github/workflows/test-with-comments.yml`
+   - View progress at: https://github.com/testdriverai/testdriverai/actions
+   - After ~30-60 seconds, you'll see a comment on your PR with:
+     - ✅ Test results summary
+     - 🎥 Embedded dashcam GIF replay
+     - 📊 Test statistics
+     - 📋 Link to full test run
+
+## What the Test Does
+
+The workflow runs `examples/assert.test.mjs` which:
+- Provisions a Chrome browser
+- Navigates to https://saucedemo.com
+- Performs login actions
+- Uses TestDriver's `assert()` to verify elements
+- Records a dashcam replay of the entire test
+- Posts results to your PR automatically
+
+## Expected Comment Output
+
+You should see a comment like:
+
+```markdown
+# 🟢 TestDriver Test Results
+
+**Status:** ✅ PASSED
+**Duration:** 25.3s
+**Platform:** linux
+**Branch:** `test-github-comments`
+**Commit:** `abc1234`
+
+## 📊 Test Summary
+
+Total:   1
+Passed:  1 ✅
+Failed:  0 ❌
+Skipped: 0 ⏭️
+
+### [📋 View Full Test Run](https://console.testdriver.ai/runs/...)
+
+## 📝 Test Results
+
+| Status | Test | File | Duration | Replay |
+|--------|------|------|----------|--------|
+| ✅ | Assert Test | `examples/assert.test.mjs` | 25.3s | [🎥 View](https://console.testdriver.ai/replay/...) |
+
+## 🎥 Dashcam Replays
+
+### Assert Test
+
+[![Assert Test](https://console.testdriver.ai/api/replay/.../gif)](https://console.testdriver.ai/replay/...)
+
+[🎬 View Full Replay](https://console.testdriver.ai/replay/...)
+```
+
+## Troubleshooting
+
+### No comment appears
+- Check Actions tab for errors: https://github.com/testdriverai/testdriverai/actions
+- Verify `TD_API_KEY` secret is set correctly
+- Ensure workflow has `pull-requests: write` permission (it does in the provided workflow)
+
+### Workflow doesn't run
+- Make sure the workflow file is on the branch you're testing
+- Check that PR is from the same repository (not a fork)
+
+### Test fails
+- View the workflow logs for details
+- Check TestDriver API status
+- Verify your API key is valid
+
+## Manual Trigger
+
+You can also trigger the workflow manually:
+1. Go to: https://github.com/testdriverai/testdriverai/actions/workflows/test-with-comments.yml
+2. Click "Run workflow"
+3. Select your branch
+4. Click "Run workflow"
+
+Note: Manual triggers won't post PR comments (no PR context), but will still run tests.
+
+## Clean Up
+
+After testing, delete the test branch:
+```bash
+gh pr close YOUR_PR_NUMBER
+git checkout main
+git branch -D test-github-comments
+git push origin --delete test-github-comments
+rm test-comment.md  # if it exists on main
+```
+
+## Next Steps
+
+Once you've confirmed it works:
+1. Customize the workflow for your needs
+2. Add more test files to the run command
+3. Set up for all PRs in your repository
+4. Share with your team!

package/docs/_scripts/generate-skills.js

@@ -0,0 +1,149 @@
+#!/usr/bin/env node
+
+const fs = require("fs");
+const path = require("path");
+
+/**
+ * Script to generate MCP skills from docs/v7/*.mdx files
+ *
+ * This script reads the frontmatter from each mdx file and generates
+ * SKILL.md files in the skills/ output directory.
+ */
+
+const DOCS_DIR = path.join(__dirname, "../v7");
+const OUTPUT_DIR = path.join(__dirname, "../../skills");
+
+// Parse YAML frontmatter from mdx content
+function parseFrontmatter(content) {
+  const frontmatterMatch = content.match(/^---\n([\s\S]*?)\n---/);
+  if (!frontmatterMatch) {
+    return { frontmatter: {}, body: content };
+  }
+
+  const frontmatterText = frontmatterMatch[1];
+  const body = content.slice(frontmatterMatch[0].length).trim();
+
+  // Simple YAML parser for our use case
+  const frontmatter = {};
+  const lines = frontmatterText.split("\n");
+  for (const line of lines) {
+    const match = line.match(/^(\w+):\s*"?([^"]*)"?\s*$/);
+    if (match) {
+      frontmatter[match[1]] = match[2];
+    }
+  }
+
+  return { frontmatter, body };
+}
+
+// Get skill name from filename
+function getSkillName(filename) {
+  return `testdriver:${filename.replace(".mdx", "")}`;
+}
+
+// Generate SKILL.md content
+function generateSkillContent(filename, frontmatter, body) {
+  const skillName = getSkillName(filename);
+  const description = frontmatter.description || frontmatter.sidebarTitle || filename.replace(".mdx", "");
+  
+  return `---
+name: ${skillName}
+description: ${description}
+---
+<!-- Generated from ${filename}. DO NOT EDIT. -->
+
+${body}
+`;
+}
+
+// Process all mdx files
+function processFiles() {
+  // Ensure output directory exists
+  if (!fs.existsSync(OUTPUT_DIR)) {
+    fs.mkdirSync(OUTPUT_DIR, { recursive: true });
+  }
+
+  const files = fs.readdirSync(DOCS_DIR);
+  const mdxFiles = files.filter(
+    (file) => file.endsWith(".mdx") && !file.startsWith("_")
+  );
+
+  console.log(`Found ${mdxFiles.length} mdx files to process\n`);
+
+  let generated = 0;
+  let errors = 0;
+
+  for (const file of mdxFiles) {
+    const filePath = path.join(DOCS_DIR, file);
+    const skillName = getSkillName(file);
+    const outputDir = path.join(OUTPUT_DIR, skillName);
+    const outputPath = path.join(outputDir, "SKILL.md");
+
+    try {
+      const content = fs.readFileSync(filePath, "utf-8");
+      const { frontmatter, body } = parseFrontmatter(content);
+      const skillContent = generateSkillContent(file, frontmatter, body);
+
+      // Create skill directory
+      if (!fs.existsSync(outputDir)) {
+        fs.mkdirSync(outputDir, { recursive: true });
+      }
+
+      fs.writeFileSync(outputPath, skillContent, "utf-8");
+      console.log(`✅ ${skillName}`);
+      generated++;
+    } catch (error) {
+      console.error(`❌ ${file}: ${error.message}`);
+      errors++;
+    }
+  }
+
+  return { generated, errors };
+}
+
+// Main function
+function main() {
+  console.log("🚀 Generating MCP skills from docs/v7/*.mdx files...\n");
+  console.log(`📂 Source: ${DOCS_DIR}`);
+  console.log(`📂 Output: ${OUTPUT_DIR}\n`);
+
+  const { generated, errors } = processFiles();
+
+  console.log(`\n✨ Complete!`);
+  console.log(`   Generated: ${generated} skills`);
+  if (errors > 0) {
+    console.log(`   Errors: ${errors}`);
+  }
+  console.log(`\nSkills written to: ${OUTPUT_DIR}`);
+}
+
+// Handle command line arguments
+if (process.argv.includes("--help") || process.argv.includes("-h")) {
+  console.log(`
+MCP Skills Generator
+
+Usage: node generate-skills.js [options]
+
+Options:
+  --help, -h    Show this help message
+
+Description:
+  This script generates MCP skill files from the docs/v7/*.mdx documentation.
+  
+  Each .mdx file is converted to a SKILL.md file with:
+  - name: testdriver:<filename>
+  - description: from frontmatter
+  - Content: the mdx body
+  
+  Output is written to the skills/ directory.
+`);
+  process.exit(0);
+}
+
+// Run the script
+try {
+  main();
+} catch (error) {
+  console.error("❌ Error:", error.message);
+  process.exit(1);
+}

package/docs/_scripts/link-replacer.js

@@ -0,0 +1,164 @@
+#!/usr/bin/env node
+
+const fs = require("fs");
+const path = require("path");
+
+/**
+ * Script to replace command references in Mintlify docs with proper links
+ *
+ * This script finds instances of command names like `hover-text`, `wait-for-text`, etc.
+ * and replaces them with proper Mintlify links to their documentation pages.
+ */
+
+// Get all command files from the commands directory
+function getCommandNames() {
+  const commandsDir = path.join(__dirname, "../", "commands");
+
+  if (!fs.existsSync(commandsDir)) {
+    console.error("Commands directory not found:", commandsDir);
+    process.exit(1);
+  }
+
+  const files = fs.readdirSync(commandsDir);
+  const commands = files
+    .filter((file) => file.endsWith(".mdx"))
+    .map((file) => file.replace(".mdx", ""));
+
+  console.log("Found commands:", commands);
+  return commands;
+}
+
+// Process a single file
+function processFile(filePath, commands) {
+  const content = fs.readFileSync(filePath, "utf-8");
+  let modified = content;
+  let changes = 0;
+
+  commands.forEach((command) => {
+    // Create regex pattern to match `command-name` but not already linked commands
+    // This pattern looks for backtick-wrapped command names that are NOT:
+    // 1. Already inside a link: [` or preceded by ]( or ](/
+    // 2. Part of an existing link structure
+    const pattern = new RegExp(
+      `(?<!\\[)\`(${command.replace(/[-/\\^$*+?.()|[\]{}]/g, "\\$&")})\`(?!\\]\\(|\\)\\]|\\]\\(/commands/)`,
+      "g",
+    );
+
+    const replacement = `[\`$1\`](/commands/$1)`;
+    const newContent = modified.replace(pattern, replacement);
+
+    if (newContent !== modified) {
+      const matchCount = (modified.match(pattern) || []).length;
+      console.log(`  - Replaced ${matchCount} instances of \`${command}\``);
+      changes += matchCount;
+      modified = newContent;
+    }
+  });
+
+  return { content: modified, changes };
+}
+
+// Process all .mdx files in a directory recursively
+function processDirectory(dirPath, commands, exclude = []) {
+  const items = fs.readdirSync(dirPath);
+  let totalChanges = 0;
+
+  items.forEach((item) => {
+    const itemPath = path.join(dirPath, item);
+    const relativePath = path.relative(process.cwd(), itemPath);
+
+    // Skip excluded directories
+    if (exclude.some((ex) => relativePath.includes(ex))) {
+      return;
+    }
+
+    const stat = fs.statSync(itemPath);
+
+    if (stat.isDirectory()) {
+      totalChanges += processDirectory(itemPath, commands, exclude);
+    } else if (item.endsWith(".mdx")) {
+      console.log(`Processing: ${relativePath}`);
+      const result = processFile(itemPath, commands);
+
+      if (result.changes > 0) {
+        fs.writeFileSync(itemPath, result.content, "utf-8");
+        console.log(`  ✅ Updated with ${result.changes} changes`);
+        totalChanges += result.changes;
+      } else {
+        console.log(`  ⏭️  No changes needed`);
+      }
+    }
+  });
+
+  return totalChanges;
+}
+
+// Main function
+function main() {
+  console.log("🚀 Starting TestDriver AI docs link replacer...\n");
+
+  // Get command names from the commands directory
+  const commands = getCommandNames();
+
+  if (commands.length === 0) {
+    console.log("No commands found. Exiting.");
+    return;
+  }
+
+  console.log(`\n📝 Processing documentation files...\n`);
+
+  // Process all .mdx files in the docs directory, excluding the commands directory itself
+  const docsDir = path.join(__dirname, "../");
+  const exclude = ["commands"]; // Don't modify the command docs themselves
+
+  const totalChanges = processDirectory(docsDir, commands, exclude);
+
+  console.log(`\n✨ Complete! Made ${totalChanges} total replacements.`);
+
+  if (totalChanges > 0) {
+    console.log("\n📋 Summary:");
+    console.log(
+      "- Command references like `hover-text` have been replaced with [`hover-text`](/commands/hover-text)",
+    );
+    console.log("- Only changed files that needed updates");
+    console.log(
+      "- Excluded the /commands directory to avoid self-referencing issues",
+    );
+    console.log("\nYou may want to review the changes before committing.");
+  } else {
+    console.log("\n🎉 All files are already properly linked!");
+  }
+}
+
+// Handle command line arguments
+if (process.argv.includes("--help") || process.argv.includes("-h")) {
+  console.log(`
+TestDriver AI Documentation Link Replacer
+
+Usage: node link-replacer.js [options]
+
+Options:
+  --help, -h    Show this help message
+  --dry-run     Show what would be changed without making actual changes (TODO)
+
+Description:
+  This script finds command references like \`hover-text\` in your Mintlify 
+  documentation and replaces them with proper links to the command reference
+  pages like [\`hover-text\`](/commands/hover-text).
+
+  The script:
+  - Scans all .mdx files in the /docs directory
+  - Excludes the /commands directory to avoid self-references  
+  - Only replaces \`command-name\` patterns that aren't already linked
+  - Preserves existing links and formatting
+`);
+  process.exit(0);
+}
+
+// Run the script
+try {
+  main();
+} catch (error) {
+  console.error("❌ Error:", error.message);
+  process.exit(1);
+}