testdriverai 7.2.25 → 7.2.27

package/docs/TEST-GITHUB-COMMENTS.md

@@ -0,0 +1,129 @@
+# Testing GitHub Comments Feature
+
+## Quick Test Instructions
+
+1. **Ensure TD_API_KEY is set**
+   - Go to: https://github.com/testdriverai/testdriverai/settings/secrets/actions
+   - Add secret named `TD_API_KEY` with your API key from https://console.testdriver.ai/team
+   - If already set, you're good to go!
+
+2. **Create a test branch and PR**
+   ```bash
+   cd /Users/ianjennings/Development/testdriverai
+   git checkout -b test-github-comments
+   
+   # Make a small change to trigger the workflow
+   echo "# Test GitHub Comments" >> test-comment.md
+   git add test-comment.md
+   git commit -m "test: trigger GitHub comment workflow"
+   git push origin test-github-comments
+   ```
+
+3. **Create Pull Request**
+   ```bash
+   gh pr create --title "Test: GitHub Comments Feature" \
+                --body "Testing automatic GitHub comments with test results and dashcam replays"
+   ```
+   
+   Or create via web: https://github.com/testdriverai/testdriverai/compare
+
+4. **Watch the Magic Happen! ✨**
+   - The workflow will automatically run: `.github/workflows/test-with-comments.yml`
+   - View progress at: https://github.com/testdriverai/testdriverai/actions
+   - After ~30-60 seconds, you'll see a comment on your PR with:
+     - ✅ Test results summary
+     - 🎥 Embedded dashcam GIF replay
+     - 📊 Test statistics
+     - 📋 Link to full test run
+
+## What the Test Does
+
+The workflow runs `test/testdriver/assert.test.mjs` which:
+- Provisions a Chrome browser
+- Navigates to https://saucedemo.com
+- Performs login actions
+- Uses TestDriver's `assert()` to verify elements
+- Records a dashcam replay of the entire test
+- Posts results to your PR automatically
+
+## Expected Comment Output
+
+You should see a comment like:
+
+```markdown
+# 🟢 TestDriver Test Results
+
+**Status:** ✅ PASSED
+**Duration:** 25.3s
+**Platform:** linux
+**Branch:** `test-github-comments`
+**Commit:** `abc1234`
+
+## 📊 Test Summary
+
+Total:   1
+Passed:  1 ✅
+Failed:  0 ❌
+Skipped: 0 ⏭️
+
+### [📋 View Full Test Run](https://console.testdriver.ai/runs/...)
+
+## 📝 Test Results
+
+| Status | Test | File | Duration | Replay |
+|--------|------|------|----------|--------|
+| ✅ | Assert Test | `test/testdriver/assert.test.mjs` | 25.3s | [🎥 View](https://console.testdriver.ai/replay/...) |
+
+## 🎥 Dashcam Replays
+
+### Assert Test
+
+[![Assert Test](https://console.testdriver.ai/api/replay/.../gif)](https://console.testdriver.ai/replay/...)
+
+[🎬 View Full Replay](https://console.testdriver.ai/replay/...)
+```
+
+## Troubleshooting
+
+### No comment appears
+- Check Actions tab for errors: https://github.com/testdriverai/testdriverai/actions
+- Verify `TD_API_KEY` secret is set correctly
+- Ensure workflow has `pull-requests: write` permission (it does in the provided workflow)
+
+### Workflow doesn't run
+- Make sure the workflow file is on the branch you're testing
+- Check that PR is from the same repository (not a fork)
+
+### Test fails
+- View the workflow logs for details
+- Check TestDriver API status
+- Verify your API key is valid
+
+## Manual Trigger
+
+You can also trigger the workflow manually:
+1. Go to: https://github.com/testdriverai/testdriverai/actions/workflows/test-with-comments.yml
+2. Click "Run workflow"
+3. Select your branch
+4. Click "Run workflow"
+
+Note: Manual triggers won't post PR comments (no PR context), but will still run tests.
+
+## Clean Up
+
+After testing, delete the test branch:
+```bash
+gh pr close YOUR_PR_NUMBER
+git checkout main
+git branch -D test-github-comments
+git push origin --delete test-github-comments
+rm test-comment.md  # if it exists on main
+```
+
+## Next Steps
+
+Once you've confirmed it works:
+1. Customize the workflow for your needs
+2. Add more test files to the run command
+3. Set up for all PRs in your repository
+4. Share with your team!

package/docs/github-integration-setup.md

@@ -0,0 +1,266 @@
+# GitHub Integration Setup
+
+TestDriver automatically posts beautiful test result comments to GitHub Pull Requests, including:
+- ✅ Test results summary with pass/fail counts
+- 🎥 Dashcam GIF replays embedded directly in comments
+- 📊 Detailed test statistics and duration
+- 🔴 Exception details with stack traces
+- 📋 Links to full test runs
+
+This guide explains how to configure GitHub authentication so TestDriver can post these comments.
+
+## How It Works
+
+TestDriver's Vitest plugin automatically detects GitHub context and posts comments when tests complete. Here's what it needs:
+
+### Required Environment Variables
+
+1. **GITHUB_TOKEN** or **GH_TOKEN** - GitHub Personal Access Token for authentication
+2. **GITHUB_REPOSITORY** - Repository in `owner/repo` format (auto-detected in GitHub Actions)
+3. **GITHUB_PR_NUMBER** - Pull Request number (auto-detected in GitHub Actions)
+4. **GITHUB_SHA** - Commit SHA (optional, auto-detected in GitHub Actions)
+
+### Auto-Detection in GitHub Actions
+
+When running in GitHub Actions, most variables are automatically available:
+- `GITHUB_REPOSITORY` - Set automatically
+- `GITHUB_SHA` - Set automatically  
+- `GITHUB_REF` - Used to detect PR number
+- Pull request info extracted from event data
+
+You only need to configure the token permissions!
+
+### Method 1: GitHub Actions (Recommended)
+
+GitHub Actions provides automatic authentication. TestDriver will detect the context automatically.
+
+**Required Permissions:**
+
+In your workflow file, use the built-in `GITHUB_TOKEN` with write permissions:
+
+```yaml
+name: TestDriver Tests
+
+on:
+  pull_request:
+    types: [opened, synchronize]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write  # Required to post comments
+      
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          
+      - name: Install dependencies
+        run: npm ci
+        
+      - name: Run TestDriver tests
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # Automatically provided
+        run: npm run test
+```
+
+**What happens automatically:**
+- `GITHUB_TOKEN`: Provided by GitHub Actions
+- `GITHUB_REPOSITORY`: Automatically set (e.g., `testdriverai/testdriverai`)
+- `GITHUB_REF`: Automatically set (e.g., `refs/pull/123/merge`)
+- PR number: Extracted automatically from `GITHUB_REF` or event data
+
+### Method 2: Manual Setup (Local or Other CI)
+
+For local development or other CI systems, set these environment variables:
+
+```bash
+# .env or export in your shell
+GITHUB_TOKEN=ghp_your_personal_access_token_here
+GITHUB_PR_NUMBER=123
+# GITHUB_REPOSITORY is auto-detected from git if not set
+```
+
+**Creating a Personal Access Token:**
+
+1. Go to [GitHub Settings → Developer settings → Personal access tokens](https://github.com/settings/tokens)
+2. Click "Generate new token" → "Generate new token (classic)"
+3. Give it a descriptive name (e.g., "TestDriver CI")
+4. Select scopes:
+   - ✅ `repo` (Full control of private repositories)
+   - Or just ✅ `public_repo` (for public repositories only)
+5. Click "Generate token"
+6. Copy the token immediately (you won't see it again!)
+7. Store it securely:
+   - **In GitHub Actions**: Add as repository secret (`Settings → Secrets and variables → Actions → New repository secret`)
+   - **Locally**: Add to `.env` file (and add `.env` to `.gitignore`!)
+
+### Method 3: Other CI Systems
+
+#### CircleCI
+```yaml
+environment:
+  GITHUB_TOKEN: $GITHUB_TOKEN
+  GITHUB_REPOSITORY: $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
+  GITHUB_PR_NUMBER: $CIRCLE_PR_NUMBER
+```
+
+#### GitLab CI
+```yaml
+variables:
+  GITHUB_TOKEN: $GITHUB_TOKEN
+  GITHUB_REPOSITORY: "your-org/your-repo"
+  GITHUB_PR_NUMBER: $CI_MERGE_REQUEST_IID
+```
+
+#### Jenkins
+```groovy
+environment {
+  GITHUB_TOKEN = credentials('github-token')
+  GITHUB_REPOSITORY = "${env.GITHUB_ORG}/${env.GITHUB_REPO}"
+  GITHUB_PR_NUMBER = "${env.CHANGE_ID}"
+}
+```
+
+## Environment Variables Reference
+
+| Variable | Priority | Description | Example | Source |
+|----------|----------|-------------|---------|--------|
+| `GITHUB_TOKEN` | 1 | GitHub personal access token | `ghp_abc...` | Auto-provided in GitHub Actions |
+| `GH_TOKEN` | 2 | Alternative GitHub token | `ghp_xyz...` | Manual setup |
+| `GITHUB_REPOSITORY` | - | Repository in `owner/repo` format | `testdriverai/testdriverai` | Auto-set in GitHub Actions |
+| `GITHUB_PR_NUMBER` | - | Pull request number | `123` | Must be set manually or by CI |
+| `GITHUB_SHA` | - | Commit SHA | `abc1234def...` | Auto-set in GitHub Actions |
+| `TESTDRIVER_SKIP_GITHUB_COMMENT` | - | Disable comment posting | `true` | Set to skip comments |
+
+**Note**: Git repository info (owner/repo, branch, commit) is auto-detected from your local `.git` directory if not provided via environment variables.
+
+## Disabling GitHub Comments
+
+To disable GitHub comments:
+
+```bash
+# Method 1: Set environment variable
+TESTDRIVER_SKIP_GITHUB_COMMENT=true npm run test
+
+# Method 2: Remove/unset the GitHub token
+unset GITHUB_TOKEN
+unset GH_TOKEN
+```
+
+In GitHub Actions workflow:
+```yaml
+- name: Run tests without GitHub comments
+  env:
+    TD_API_KEY: ${{ secrets.TD_API_KEY }}
+    TESTDRIVER_SKIP_GITHUB_COMMENT: true
+  run: npm test
+```
+
+## Troubleshooting
+
+### "GitHub token not found, skipping comment posting"
+
+Check that the token is set:
+```bash
+echo $GITHUB_TOKEN  # or GH_TOKEN
+```
+
+### "Neither PR number nor commit SHA found"
+
+Set the PR number:
+```bash
+export GITHUB_PR_NUMBER=123
+```
+
+Or ensure `GITHUB_SHA` is set (auto-set in GitHub Actions).
+
+### "Repository info not available"
+
+The plugin auto-detects repo info from your `.git` directory. If that fails, set:
+```bash
+export GITHUB_REPOSITORY=owner/repo
+```
+
+### "Resource not accessible by integration"
+
+Your token doesn't have `pull-requests: write` permission. Check:
+- GitHub Actions: Add `permissions.pull-requests: write` to workflow
+- Personal token: Ensure it has `repo` or `public_repo` scope
+
+### "Not Found" or "403 Forbidden"
+
+- Token doesn't have access to the repository
+- Repository name format is incorrect (must be `owner/repo`)
+- PR number is invalid
+
+### Comments not appearing
+
+- Check that tests are running in the context of a pull request
+- Verify the token has write access
+- Look for error messages in the test output
+- Run with `DEBUG=testdriver:github` for verbose logging
+
+## Security Best Practices
+
+1. **Never commit tokens** to git repositories
+2. **Use GitHub Actions secrets** for CI/CD
+3. **Use fine-grained tokens** with minimum required permissions
+4. **Rotate tokens regularly** (at least every 90 days)
+5. **Revoke tokens** immediately if compromised
+6. **Use organization secrets** for shared tokens across repos
+
+## Example Comment Output
+
+When properly configured, TestDriver will post comments like:
+
+```markdown
+# 🟢 TestDriver Test Results
+
+**Status:** ✅ PASSED
+**Duration:** 45.23s
+**Platform:** linux
+**Branch:** `feature/new-login`
+**Commit:** `abc1234`
+
+## 📊 Test Summary
+
+Total:   3
+Passed:  3 ✅
+Failed:  0 ❌
+Skipped: 0 ⏭️
+
+### [📋 View Full Test Run](https://app.testdriver.ai/runs/...)
+
+## 📝 Test Results
+
+| Status | Test | File | Duration | Replay |
+|--------|------|------|----------|--------|
+| ✅ | should log in successfully | `login.test.mjs` | 12.34s | [🎥 View](https://app.testdriver.ai/replay/...) |
+| ✅ | should navigate to dashboard | `navigation.test.mjs` | 8.91s | [🎥 View](https://app.testdriver.ai/replay/...) |
+
+## 🎥 Dashcam Replays
+
+### should log in successfully
+
+[![Login test](https://app.testdriver.ai/api/replay/.../gif)](https://app.testdriver.ai/replay/...)
+
+[🎬 View Full Replay](https://app.testdriver.ai/replay/...)
+
+---
+<sub>Generated by [TestDriver](https://testdriver.ai) • Run ID: `run_abc123`</sub>
+```
+
+The GIF replays are automatically embedded so reviewers can see what happened without leaving GitHub!
+
+## Getting Help
+
+- 📚 [Full Documentation](https://docs.testdriver.ai)
+- 💬 [Discord Community](https://discord.gg/testdriver)
+- 🐛 [Report Issues](https://github.com/testdriverai/testdriverai/issues)

package/examples/github-actions.yml

@@ -0,0 +1,68 @@
+name: TestDriver Tests with GitHub Comments
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+  push:
+    branches: [main, develop]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    
+    permissions:
+      # Required for posting comments on PRs
+      pull-requests: write
+      # Required for posting comments on commits
+      contents: write
+    
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: 'npm'
+      
+      - name: Install dependencies
+        run: npm ci
+      
+      - name: Run TestDriver tests
+        env:
+          # TestDriver API key (add this to your repository secrets)
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          
+          # GitHub token for posting comments (automatically provided)
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          
+          # PR number for PR comments
+          GITHUB_PR_NUMBER: ${{ github.event.pull_request.number }}
+          
+          # Git information (automatically available)
+          # GITHUB_SHA, GITHUB_REF_NAME, GITHUB_REPOSITORY, GITHUB_ACTOR
+          
+        run: npm run test:sdk
+        
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-results
+          path: |
+            test-report.junit.xml
+            test-results/
+          retention-days: 30
+      
+      - name: Comment PR on failure
+        if: failure() && github.event_name == 'pull_request'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: '❌ TestDriver tests failed. Check the workflow logs for details.'
+            })

package/examples/github-comment-demo.test.mjs

@@ -0,0 +1,42 @@
+/**
+ * Example test demonstrating GitHub comment integration
+ * 
+ * When run in CI with proper environment variables set, this will:
+ * 1. Run the tests
+ * 2. Record dashcam replays
+ * 3. Post a beautiful GitHub comment with results
+ */
+
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "../lib/vitest/hooks.mjs";
+
+describe("GitHub Comment Demo", () => {
+  it("should pass with dashcam replay", async (context) => {
+    const testdriver = TestDriver(context, { headless: true });
+    
+    await testdriver.provision.chrome({
+      url: 'https://www.example.com',
+    });
+
+    const heading = await testdriver.find("heading with Example Domain");
+    const result = await testdriver.assert("I can see 'Example Domain' heading");
+    expect(result).toBeTruthy();
+  });
+
+  it("should demonstrate failure handling", async (context) => {
+    const testdriver = TestDriver(context, { headless: true });
+    
+    await testdriver.provision.chrome({
+      url: 'https://www.example.com',
+    });
+
+    // This will fail intentionally to show error reporting
+    const result = await testdriver.assert("I can see a navigation menu with 20 items");
+    expect(result).toBeTruthy(); // This will fail
+  });
+
+  it("should skip when needed", async (context) => {
+    // Skipped tests show in the summary
+    context.skip();
+  });
+});

package/interfaces/vitest-plugin.mjs

@@ -3,6 +3,7 @@ import crypto from "crypto";
 import { createRequire } from "module";
 import path from "path";
 import { setTestRunInfo } from "./shared-test-state.mjs";
+import { generateGitHubComment, postOrUpdateTestResults } from "../lib/github-comment.mjs";
 
 // Use createRequire to import CommonJS modules without esbuild processing
 const require = createRequire(import.meta.url);
@@ -132,6 +133,7 @@ export const pluginState = {
   client: null,
   startTime: null,
   testCases: new Map(),
+  recordedTestCases: [], // Store recorded test case data for GitHub comment
   token: null,
   detectedPlatform: null,
   pendingTestCaseRecords: new Set(),
@@ -658,6 +660,9 @@ class TestDriverReporter {
         logger.debug(`🔗 View test run: ${testRunUrl}`);
         // Output in a parseable format for CI
         console.log(`TESTDRIVER_RUN_URL=${testRunUrl}`);
+        
+        // Post GitHub comment if in CI environment
+        await postGitHubCommentIfEnabled(testRunUrl, stats, completeData);
       }
 
       logger.debug(`✅ Test run completed: ${stats.passedTests}/${stats.totalTests} passed`);
@@ -793,6 +798,12 @@ class TestDriverReporter {
       const testCaseDbId = testCaseResponse.data?.id;
       const testRunDbId = process.env.TD_TEST_RUN_DB_ID;
 
+      // Store test case data for GitHub comment generation
+      pluginState.recordedTestCases.push({
+        ...testCaseData,
+        id: testCaseDbId,
+      });
+
       console.log('');
       console.log(`🔗 Test Report: ${getConsoleUrl(pluginState.apiRoot)}/runs/${testRunDbId}/${testCaseDbId}`);
     } catch (error) {
@@ -977,6 +988,95 @@ function getGitInfo() {
   return info;
 }
 
+// ============================================================================
+// GitHub Comment Helper
+// ============================================================================
+
+/**
+ * Post GitHub comment with test results if enabled
+ * Checks for GitHub token and PR number in environment variables
+ * @param {string} testRunUrl - URL to the test run
+ * @param {Object} stats - Test statistics
+ * @param {Object} completeData - Test run completion data
+ */
+async function postGitHubCommentIfEnabled(testRunUrl, stats, completeData) {
+  try {
+    // Check if GitHub comments are explicitly disabled
+    if (process.env.TESTDRIVER_SKIP_GITHUB_COMMENT === 'true') {
+      logger.debug('GitHub comments disabled via TESTDRIVER_SKIP_GITHUB_COMMENT');
+      return;
+    }
+    
+    // Check if GitHub comment posting is enabled
+    const githubToken = process.env.GITHUB_TOKEN || process.env.GH_TOKEN;
+    const prNumber = process.env.GITHUB_PR_NUMBER;
+    const commitSha = process.env.GITHUB_SHA || pluginState.gitInfo.commit;
+    
+    // Only post if we have a token and either a PR number or commit SHA
+    if (!githubToken) {
+      logger.debug('GitHub token not found, skipping comment posting');
+      return;
+    }
+    
+    if (!prNumber && !commitSha) {
+      logger.debug('Neither PR number nor commit SHA found, skipping comment posting');
+      return;
+    }
+    
+    // Extract owner/repo from git info
+    const repo = pluginState.gitInfo.repo;
+    if (!repo) {
+      logger.warn('Repository info not available, skipping GitHub comment');
+      return;
+    }
+    
+    const [owner, repoName] = repo.split('/');
+    if (!owner || !repoName) {
+      logger.warn('Invalid repository format, expected owner/repo');
+      return;
+    }
+    
+    logger.debug('Preparing GitHub comment...');
+    
+    // Prepare test run data for comment
+    const testRunData = {
+      runId: pluginState.testRunId,
+      status: completeData.status,
+      totalTests: stats.totalTests,
+      passedTests: stats.passedTests,
+      failedTests: stats.failedTests,
+      skippedTests: stats.skippedTests,
+      duration: completeData.duration,
+      testRunUrl,
+      platform: completeData.platform || pluginState.detectedPlatform || 'unknown',
+      branch: pluginState.gitInfo.branch || 'unknown',
+      commit: commitSha || 'unknown',
+    };
+    
+    // Use recorded test cases from pluginState
+    const testCases = pluginState.recordedTestCases || [];
+    
+    logger.info(`Posting GitHub comment with ${testCases.length} test cases...`);
+    
+    // Post or update GitHub comment
+    const githubOptions = {
+      token: githubToken,
+      owner,
+      repo: repoName,
+      prNumber: prNumber ? parseInt(prNumber, 10) : undefined,
+      commitSha: commitSha,
+    };
+    
+    const comment = await postOrUpdateTestResults(testRunData, testCases, githubOptions);
+    logger.info(`✅ GitHub comment posted: ${comment.html_url}`);
+    console.log(`\n🔗 GitHub Comment: ${comment.html_url}\n`);
+    
+  } catch (error) {
+    logger.warn('Failed to post GitHub comment:', error.message);
+    logger.debug('GitHub comment error stack:', error.stack);
+  }
+}
+
 // ============================================================================
 // API Methods
 // ============================================================================