testdriverai 7.2.25 → 7.2.27

package/.github/workflows/test-with-comments.yml

@@ -0,0 +1,73 @@
+name: TestDriver Tests with GitHub Comments
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+  push:
+    branches: [main]
+  workflow_dispatch: # Allows manual trigger from Actions tab
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    
+    permissions:
+      contents: read
+      pull-requests: write  # Required to post comments on PRs
+    
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: 'npm'
+      
+      - name: Install dependencies
+        run: npm ci
+      
+      - name: Run assert test with GitHub comments
+        env:
+          # TestDriver API key (from repository secrets)
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          
+          # GitHub token for posting comments (auto-provided)
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          
+          # PR number for PR comments (auto-extracted for pull_request events)
+          GITHUB_PR_NUMBER: ${{ github.event.pull_request.number }}
+          
+          # Git information (automatically provided by GitHub Actions)
+          # GITHUB_SHA, GITHUB_REF, GITHUB_REPOSITORY are already set
+          
+        run: |
+          echo "🚀 Running TestDriver assert test..."
+          echo "📍 Repository: $GITHUB_REPOSITORY"
+          echo "🔢 PR Number: ${{ github.event.pull_request.number || 'N/A (not a PR)' }}"
+          echo "📦 Running test..."
+          npm run test:sdk -- test/testdriver/assert.test.mjs
+        
+      - name: Upload test results (on failure)
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-results
+          path: |
+            test-report.junit.xml
+            .testdriver/
+          retention-days: 7
+      
+      - name: Comment on PR (manual fallback if auto-comment fails)
+        if: failure() && github.event_name == 'pull_request'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const testRunUrl = process.env.TESTDRIVER_RUN_URL || 'Check workflow logs';
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: `⚠️ TestDriver tests encountered an error. ${testRunUrl}`
+            })

package/docs/GITHUB_COMMENTS.md

@@ -0,0 +1,330 @@
+# GitHub Comment Integration
+
+TestDriver can automatically post beautiful, formatted comments to your GitHub pull requests and commits with test results, including dashcam replays, exceptions, and detailed statistics.
+
+## Features
+
+- 📊 **Test Results Summary** - Pass/fail statistics and duration
+- 🎥 **Dashcam GIF Replays** - Embedded GIF previews with links to full replays
+- ❌ **Exception Details** - Error messages and stack traces for failed tests
+- 📋 **Detailed Test Table** - Status, file, duration, and replay link for each test
+- 🔄 **Auto-update** - Updates existing comment instead of creating duplicates
+- 🎨 **Beautiful Formatting** - Professional markdown with emojis and badges
+
+## Setup
+
+### 1. Enable in CI Environment
+
+Set the required environment variables in your CI workflow:
+
+#### GitHub Actions
+
+```yaml
+name: TestDriver Tests
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      
+      - name: Setup Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: '20'
+      
+      - name: Install dependencies
+        run: npm install
+      
+      - name: Run TestDriver tests
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: npm run test:sdk
+```
+
+**Important:** For pull request events, `GITHUB_PR_NUMBER` must be explicitly set from `github.event.pull_request.number`.
+
+### 2. Environment Variables
+
+| Variable | Description | Required | Example |
+|----------|-------------|----------|---------|
+| `TD_API_KEY` | TestDriver API key | ✅ Yes | Get from [console.testdriver.ai](https://console.testdriver.ai/team) |
+| `GITHUB_TOKEN` | GitHub token with PR write permissions | ✅ Yes | Automatically available in GitHub Actions |
+| `GITHUB_PR_NUMBER` | Pull request number | For PR comments | `123` |
+| `GITHUB_SHA` | Commit SHA | For commit comments | Automatically set in GitHub Actions |
+
+**Token Alternatives:**
+- `GITHUB_TOKEN` (preferred)
+- `GH_TOKEN` (alternative)
+
+### 3. Comment Destinations
+
+TestDriver will post comments based on available context:
+
+- **Pull Request Comment** - If `GITHUB_PR_NUMBER` is set (recommended)
+- **Commit Comment** - If only `GITHUB_SHA` is available (fallback)
+
+## Example Comment
+
+Here's what a TestDriver GitHub comment looks like:
+
+---
+
+# 🟢 TestDriver Test Results
+
+**Status:** ✅ PASSED  
+**Duration:** 2m 34s  
+**Platform:** linux  
+**Branch:** `feature/new-test`  
+**Commit:** `a1b2c3d`
+
+## 📊 Test Summary
+
+```
+Total:   5
+Passed:  4 ✅
+Failed:  1 ❌
+Skipped: 0 ⏭️
+```
+
+### [📋 View Full Test Run](https://console.testdriver.ai/runs/123)
+
+## 📝 Test Results
+
+| Status | Test | File | Duration | Replay |
+|--------|------|------|----------|--------|
+| ✅ | should login successfully | `tests/auth.test.mjs` | 12.5s | [🎥 View](https://console.testdriver.ai/replay/abc123) |
+| ✅ | should add item to cart | `tests/cart.test.mjs` | 8.2s | [🎥 View](https://console.testdriver.ai/replay/def456) |
+| ❌ | should complete checkout | `tests/checkout.test.mjs` | 15.3s | [🎥 View](https://console.testdriver.ai/replay/ghi789) |
+
+## 🎥 Dashcam Replays
+
+### should login successfully
+
+[![should login successfully](https://console.testdriver.ai/api/replay/abc123/gif)](https://console.testdriver.ai/replay/abc123)
+
+[🎬 View Full Replay](https://console.testdriver.ai/replay/abc123)
+
+## 🔴 Failures
+
+### should complete checkout
+
+**File:** `tests/checkout.test.mjs`
+
+**📹 [Watch Replay](https://console.testdriver.ai/replay/ghi789)**
+
+```
+AssertionError: expected false to be truthy
+```
+
+<details>
+<summary>Stack Trace</summary>
+
+```
+AssertionError: expected false to be truthy
+    at /workspace/tests/checkout.test.mjs:45:7
+    ...
+```
+</details>
+
+---
+<sub>Generated by [TestDriver](https://testdriver.ai) • Run ID: `1234567890-abc123`</sub>
+
+---
+
+## Configuration
+
+### Customizing Comment Behavior
+
+You can control GitHub comment posting with environment variables:
+
+```bash
+# Disable GitHub comments entirely
+TESTDRIVER_SKIP_GITHUB_COMMENT=true npm run test
+
+# Use custom GitHub token
+GH_TOKEN=ghp_yourtoken npm run test
+```
+
+### GitLab / Other CI Systems
+
+While the integration is optimized for GitHub, you can use the comment generator programmatically:
+
+```javascript
+import { generateGitHubComment } from 'testdriverai/lib/github-comment.mjs';
+
+const testRunData = {
+  runId: '1234567890-abc',
+  status: 'passed',
+  totalTests: 5,
+  passedTests: 4,
+  failedTests: 1,
+  skippedTests: 0,
+  duration: 154000,
+  testRunUrl: 'https://console.testdriver.ai/runs/123',
+  platform: 'linux',
+  branch: 'main',
+  commit: 'a1b2c3d',
+};
+
+const testCases = [
+  {
+    testName: 'should work',
+    testFile: 'test.test.mjs',
+    status: 'passed',
+    duration: 5000,
+    replayUrl: 'https://console.testdriver.ai/replay/abc',
+  },
+  // ... more tests
+];
+
+const markdown = generateGitHubComment(testRunData, testCases);
+console.log(markdown);
+```
+
+## Troubleshooting
+
+### Comment not posting
+
+1. **Check environment variables:**
+   ```bash
+   echo $GITHUB_TOKEN
+   echo $GITHUB_PR_NUMBER
+   ```
+
+2. **Check CI logs:**
+   Look for log messages:
+   - `"GitHub token not found, skipping comment posting"`
+   - `"Repository info not available"`
+   - `"Posted GitHub comment"`
+
+3. **Verify token permissions:**
+   - GitHub Actions: `GITHUB_TOKEN` should have `write` permissions for `pull-requests` or `contents`
+   - Personal tokens: Needs `repo` scope
+
+### Comment not updating
+
+If you see duplicate comments instead of updates:
+- Ensure the bot/user posting comments is consistent across runs
+- The update mechanism looks for existing comments with "Generated by [TestDriver]" signature
+
+### Replays not showing
+
+If dashcam replays aren't included:
+- Verify dashcam is starting correctly (check for `🎥 Dashcam URL:` in logs)
+- Ensure tests are calling `await testdriver.provision.*()` which auto-starts dashcam
+- Check that dashcam stops successfully (look for replay URLs in output)
+
+### GIF embeds not loading
+
+The GIF embeds are generated dynamically by the TestDriver API:
+- URL format: `https://console.testdriver.ai/api/replay/{replayId}/gif`
+- If GIFs don't load, the API endpoint may be unavailable
+- Fallback: Users can still click the "View Full Replay" link
+
+## API Reference
+
+### `generateGitHubComment(testRunData, testCases)`
+
+Generates markdown for a GitHub comment.
+
+**Parameters:**
+- `testRunData` (Object): Test run summary data
+  - `runId` (string): Unique test run ID
+  - `status` (string): Overall status ('passed', 'failed', 'cancelled')
+  - `totalTests` (number): Total number of tests
+  - `passedTests` (number): Number of passed tests
+  - `failedTests` (number): Number of failed tests
+  - `skippedTests` (number): Number of skipped tests
+  - `duration` (number): Total duration in milliseconds
+  - `testRunUrl` (string): URL to full test run
+  - `platform` (string): Test platform ('linux', 'windows', 'mac')
+  - `branch` (string): Git branch name
+  - `commit` (string): Git commit SHA
+- `testCases` (Array): Array of test case objects
+  - `testName` (string): Test name
+  - `testFile` (string): Test file path
+  - `status` (string): Test status
+  - `duration` (number): Test duration in ms
+  - `replayUrl` (string): Dashcam replay URL (optional)
+  - `errorMessage` (string): Error message if failed (optional)
+  - `errorStack` (string): Stack trace if failed (optional)
+
+**Returns:** String (markdown)
+
+### `postOrUpdateTestResults(testRunData, testCases, githubOptions)`
+
+Posts or updates a GitHub comment with test results.
+
+**Parameters:**
+- `testRunData` (Object): Same as `generateGitHubComment`
+- `testCases` (Array): Same as `generateGitHubComment`
+- `githubOptions` (Object): GitHub API configuration
+  - `token` (string): GitHub token
+  - `owner` (string): Repository owner
+  - `repo` (string): Repository name
+  - `prNumber` (number): PR number (optional)
+  - `commitSha` (string): Commit SHA (optional)
+
+**Returns:** Promise<Object> - GitHub API response
+
+## Examples
+
+### Manual Comment Posting
+
+```javascript
+import { postOrUpdateTestResults } from 'testdriverai/lib/github-comment.mjs';
+
+await postOrUpdateTestResults(
+  {
+    runId: '123',
+    status: 'passed',
+    totalTests: 10,
+    passedTests: 10,
+    failedTests: 0,
+    skippedTests: 0,
+    duration: 60000,
+    testRunUrl: 'https://console.testdriver.ai/runs/123',
+    platform: 'linux',
+    branch: 'main',
+    commit: 'abc123',
+  },
+  testCases,
+  {
+    token: process.env.GITHUB_TOKEN,
+    owner: 'myorg',
+    repo: 'myrepo',
+    prNumber: 456,
+  }
+);
+```
+
+### Custom Formatting
+
+```javascript
+import { generateGitHubComment } from 'testdriverai/lib/github-comment.mjs';
+
+const markdown = generateGitHubComment(testRunData, testCases);
+
+// Customize markdown
+const customMarkdown = markdown.replace(
+  '# 🟢 TestDriver Test Results',
+  '# 🟢 My Custom Test Report'
+);
+
+// Post using GitHub API directly
+// ... your posting logic
+```
+
+## Support
+
+- **Documentation:** [testdriver.ai/docs](https://testdriver.ai/docs)
+- **Issues:** [github.com/testdriverai/testdriverai/issues](https://github.com/testdriverai/testdriverai/issues)
+- **Discord:** Join our [community Discord](https://discord.gg/testdriver)

package/docs/GITHUB_COMMENTS_ANNOUNCEMENT.md

@@ -0,0 +1,167 @@
+# 🎉 GitHub Comments Feature
+
+## What's New
+
+TestDriver now automatically posts beautiful, detailed comments to your GitHub pull requests and commits with:
+
+- ✅ **Test Results** - Pass/fail statistics at a glance
+- 🎥 **Dashcam GIF Replays** - Embedded animated previews of each test
+- 📊 **Detailed Statistics** - Duration, platform, branch, commit info
+- ❌ **Exception Details** - Full error messages and stack traces for failures
+- 🔄 **Smart Updates** - Updates existing comments instead of creating duplicates
+
+## Quick Start
+
+### 1. Add to GitHub Actions
+
+```yaml
+- name: Run TestDriver tests
+  env:
+    TD_API_KEY: ${{ secrets.TD_API_KEY }}
+    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    GITHUB_PR_NUMBER: ${{ github.event.pull_request.number }}
+  run: npm run test:sdk
+```
+
+### 2. That's it!
+
+TestDriver will automatically detect the CI environment and post comments when:
+- `GITHUB_TOKEN` is present
+- `GITHUB_PR_NUMBER` (for PR comments) or `GITHUB_SHA` (for commit comments) is set
+
+## Example Comment
+
+![GitHub Comment Preview](https://placeholder.com/github-comment-preview.png)
+
+Your team will see:
+- **Status badges** with pass/fail counts
+- **Embedded GIF replays** directly in the comment
+- **Clickable links** to full test runs and individual replays
+- **Collapsible error details** with full stack traces
+
+## Features
+
+### 🎥 Dashcam Replays
+
+Every test automatically records a dashcam replay. These appear in the GitHub comment as:
+1. **GIF preview** - Shows the test execution as an animated image
+2. **Link to full replay** - Opens the complete replay in TestDriver console
+3. **Test-specific context** - Each test has its own replay section
+
+### 📊 Test Statistics
+
+The comment shows:
+- Total test count
+- Pass/fail/skip breakdown
+- Total execution time
+- Platform (Linux/Windows/Mac)
+- Git branch and commit
+
+### ❌ Exception Handling
+
+Failed tests include:
+- Error message prominently displayed
+- Collapsible stack trace
+- Direct link to the failing test's replay
+- File and line number information
+
+### 🔄 Comment Updates
+
+TestDriver intelligently:
+- **Updates existing comments** on subsequent pushes to the same PR
+- **Creates new comments** only when needed
+- **Identifies its own comments** using a signature
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `TD_API_KEY` | TestDriver API key | ✅ Yes |
+| `GITHUB_TOKEN` | GitHub token | ✅ Yes |
+| `GITHUB_PR_NUMBER` | PR number for PR comments | 📝 Recommended |
+| `GITHUB_SHA` | Commit SHA (fallback) | ⚙️ Auto-detected |
+| `TESTDRIVER_SKIP_GITHUB_COMMENT` | Set to 'true' to disable | ❌ No |
+
+### Disabling Comments
+
+To disable GitHub comments for a specific run:
+
+```bash
+TESTDRIVER_SKIP_GITHUB_COMMENT=true npm run test
+```
+
+## Documentation
+
+For complete documentation, see:
+- **[GitHub Comments Guide](./GITHUB_COMMENTS.md)** - Full setup and configuration
+- **[Example Workflow](./../examples/github-actions.yml)** - Copy-paste GitHub Actions workflow
+- **[Example Test](./../examples/github-comment-demo.test.mjs)** - Demo test file
+
+## Troubleshooting
+
+### Comment not appearing?
+
+1. **Check environment variables:**
+   ```bash
+   echo "Token: ${GITHUB_TOKEN:0:10}..."
+   echo "PR: $GITHUB_PR_NUMBER"
+   ```
+
+2. **Verify permissions:**
+   - GitHub Actions needs `pull-requests: write` or `contents: write`
+
+3. **Check logs:**
+   - Look for "Posting GitHub comment..." in test output
+   - Check for "GitHub token not found" warnings
+
+### Replays not embedded?
+
+- Ensure tests are using `await testdriver.provision.*()` 
+- Check that dashcam stops successfully (look for "🎥 Dashcam URL")
+- Verify replay URLs in test output
+
+## Examples
+
+### Basic Test with Comment
+
+```javascript
+it("should login successfully", async (context) => {
+  const testdriver = TestDriver(context, { headless: true });
+  
+  await testdriver.provision.chrome({
+    url: 'https://your-app.com/login',
+  });
+
+  // Test steps...
+  const result = await testdriver.assert("I'm logged in");
+  expect(result).toBeTruthy();
+  
+  // Dashcam automatically recorded and will appear in GitHub comment!
+});
+```
+
+### Local Testing
+
+Run locally without posting comments:
+
+```bash
+npm run test:sdk
+```
+
+Run locally WITH comment posting (requires token):
+
+```bash
+GITHUB_TOKEN=ghp_xxx GITHUB_PR_NUMBER=123 npm run test:sdk
+```
+
+## Support
+
+- **Documentation:** [testdriver.ai/docs](https://testdriver.ai/docs)
+- **Issues:** [GitHub Issues](https://github.com/testdriverai/testdriverai/issues)
+- **Discord:** [Join our community](https://discord.gg/testdriver)
+
+---
+
+Made with ❤️ by the TestDriver team

package/docs/QUICK-START-GITHUB-COMMENTS.md

@@ -0,0 +1,84 @@
+# Quick Start: GitHub Comments
+
+## For GitHub Actions (Easiest!)
+
+Add this to your `.github/workflows/test.yml`:
+
+```yaml
+name: Tests
+
+on: pull_request
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write  # ← This is all you need!
+      
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Setup Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          
+      - run: npm ci
+      
+      - name: Run TestDriver tests
+        env:
+          TD_API_KEY: ${{ secrets.TD_API_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # Auto-provided
+        run: npm test
+```
+
+That's it! TestDriver will automatically:
+- ✅ Detect the PR number
+- ✅ Detect the repository
+- ✅ Post beautiful comments with test results
+- 🎥 Embed dashcam GIF replays
+- 📊 Show pass/fail statistics
+- 🔴 Display exception details
+
+## For Local Development
+
+1. **Create a GitHub Personal Access Token**
+   - Go to https://github.com/settings/tokens
+   - Click "Generate new token (classic)"
+   - Select scope: `repo` or `public_repo`
+   - Copy the token
+
+2. **Set environment variables**
+   ```bash
+   export GITHUB_TOKEN=ghp_your_token_here
+   export GITHUB_PR_NUMBER=123
+   npm test
+   ```
+
+## What Gets Posted
+
+```markdown
+# 🟢 TestDriver Test Results
+
+**Status:** ✅ PASSED
+**Duration:** 12.3s
+
+## 📝 Test Results
+| Status | Test | Duration | Replay |
+|--------|------|----------|--------|
+| ✅ | Login test | 4.5s | [🎥 View](link) |
+
+## 🎥 Dashcam Replays
+[Animated GIF embedded here]
+```
+
+## Disable Comments
+
+```bash
+TESTDRIVER_SKIP_GITHUB_COMMENT=true npm test
+```
+
+## Full Documentation
+
+See [github-integration-setup.md](./github-integration-setup.md) for complete details.