@testdriverai/agent 7.8.0-canary.10

package/docs/v7/copilot/auto-healing.mdx

@@ -0,0 +1,265 @@
+---
+title: "Auto-Healing Tests"
+sidebarTitle: "Auto-Healing"
+description: "Set up automatic test repair in CI using GitHub Copilot and TestDriver"
+icon: "bandage"
+---
+
+Auto-healing tests automatically fix themselves when your application changes. By integrating GitHub Copilot with TestDriver in your CI pipeline, you can have AI investigate test failures and propose fixes.
+
+## How It Works
+
+When a test fails in CI:
+
+1. **GitHub Actions detects the failure**
+2. **Copilot spawns the TestDriver agent** with access to the MCP server
+3. **The agent investigates** by running the failing test and analyzing what changed
+4. **Copilot creates a fix** by updating the test code
+5. **A pull request is opened** with the proposed changes for review
+
+## Setting Up Auto-Healing
+
+<Steps>
+  <Step title="Add Repository Secrets">
+    Add your TestDriver API key to your repository secrets:
+
+    1. Go to **Settings → Secrets and variables → Actions**
+    2. Click **New repository secret**
+    3. Add `TD_API_KEY` with your API key value
+  </Step>
+
+  <Step title="Create the Auto-Heal Workflow">
+    Add a GitHub Actions workflow that triggers on test failures:
+
+    ```yaml .github/workflows/auto-heal.yml
+    name: Auto-Heal Tests
+
+    on:
+      workflow_run:
+        workflows: ["Tests"]  # Your main test workflow
+        types: [completed]
+        branches: [main]
+
+    jobs:
+      auto-heal:
+        if: ${{ github.event.workflow_run.conclusion == 'failure' }}
+        runs-on: ubuntu-latest
+        permissions:
+          contents: write
+          pull-requests: write
+
+        steps:
+          - uses: actions/checkout@v4
+
+          - name: Setup Node.js
+            uses: actions/setup-node@v4
+            with:
+              node-version: "20"
+
+          - name: Install dependencies
+            run: npm ci
+
+          - name: Run failing tests and capture output
+            id: tests
+            continue-on-error: true
+            run: |
+              vitest run 2>&1 | tee test-output.txt
+              echo "output<<EOF" >> $GITHUB_OUTPUT
+              cat test-output.txt >> $GITHUB_OUTPUT
+              echo "EOF" >> $GITHUB_OUTPUT
+            env:
+              TD_API_KEY: ${{ secrets.TD_API_KEY }}
+
+          - name: Invoke Copilot to fix tests
+            uses: github/copilot-action@v1
+            with:
+              prompt: |
+                @testdriver The following tests failed:
+
+                ${{ steps.tests.outputs.output }}
+
+                Please investigate each failure by:
+                1. Running the failing test
+                2. Analyzing what changed in the UI or behavior
+                3. Updating the test code to fix the issue
+
+                Create a commit with your changes.
+            env:
+              TD_API_KEY: ${{ secrets.TD_API_KEY }}
+
+          - name: Create Pull Request
+            uses: peter-evans/create-pull-request@v5
+            with:
+              title: "Auto-heal: Fix failing tests"
+              body: |
+                This PR was automatically generated by the auto-heal workflow.
+
+                ## Changes
+                The following tests were updated to fix failures detected in CI.
+
+                ## Review
+                Please review the changes carefully before merging.
+              branch: auto-heal/${{ github.run_id }}
+              commit-message: "fix: auto-heal failing tests"
+    ```
+  </Step>
+
+  <Step title="Configure Test Workflow">
+    Make sure your main test workflow has a name that matches the `workflows` trigger:
+
+    ```yaml .github/workflows/tests.yml
+    name: Tests  # Must match the workflow_run trigger
+
+    on:
+      push:
+        branches: [main]
+      pull_request:
+
+    jobs:
+      test:
+        runs-on: ubuntu-latest
+        steps:
+          - uses: actions/checkout@v4
+          - uses: actions/setup-node@v4
+            with:
+              node-version: "20"
+          - run: npm ci
+          - run: vitest run
+            env:
+              TD_API_KEY: ${{ secrets.TD_API_KEY }}
+    ```
+  </Step>
+</Steps>
+
+## Example: Button Text Change
+
+Here's what auto-healing looks like when a button's text changes:
+
+<Steps>
+  <Step title="Application Changes">
+    A developer changes a button's text from "Submit" to "Send":
+
+    ```html
+    <!-- Before -->
+    <button>Submit</button>
+
+    <!-- After -->
+    <button>Send</button>
+    ```
+  </Step>
+
+  <Step title="Test Fails">
+    The test fails because it's looking for the old text:
+
+    ```javascript
+    // This now fails
+    const submitButton = await testdriver.find("Submit button");
+    ```
+  </Step>
+
+  <Step title="Auto-Heal Triggers">
+    The auto-heal workflow runs, and Copilot investigates:
+
+    ```
+    The test is looking for a "Submit button" but I see a "Send button"
+    on the page. The button functionality is the same, just the text changed.
+    I'll update the test to use the new text.
+    ```
+  </Step>
+
+  <Step title="PR Created">
+    A pull request is opened with the fix:
+
+    ```javascript
+    // Updated by auto-heal
+    const submitButton = await testdriver.find("Send button");
+    ```
+  </Step>
+</Steps>
+
+## Configuration Options
+
+### Selective Auto-Healing
+
+You can limit auto-healing to specific test files or patterns:
+
+```yaml
+- name: Run failing tests
+  run: |
+    # Only heal tests in the e2e directory
+    vitest run tests/e2e/ 2>&1 | tee test-output.txt
+```
+
+### Manual Approval
+
+For safety, you can require manual approval before auto-heal runs:
+
+```yaml
+jobs:
+  auto-heal:
+    environment: auto-heal  # Requires approval
+```
+
+Configure the environment in **Settings → Environments** with required reviewers.
+
+### Limiting Changes
+
+Add instructions to constrain what the AI can change:
+
+```yaml
+- name: Invoke Copilot to fix tests
+  uses: github/copilot-action@v1
+  with:
+    prompt: |
+      @testdriver Fix the failing tests.
+
+      Rules:
+      - Only update element selectors and text matching
+      - Do not change test logic or assertions
+      - Do not add or remove tests
+      - Keep changes minimal
+```
+
+## Best Practices
+
+<AccordionGroup>
+  <Accordion title="Always review auto-heal PRs">
+    Auto-heal is a tool, not a replacement for human judgment. Review all changes before merging to ensure the test still validates what you intended.
+  </Accordion>
+
+  <Accordion title="Use descriptive element descriptions">
+    Tests with clear, semantic descriptions are easier for the AI to heal:
+    ```javascript
+    // ✅ Good - describes purpose
+    await testdriver.find("primary call-to-action button in the hero section");
+
+    // ❌ Bad - too vague
+    await testdriver.find("button");
+    ```
+  </Accordion>
+
+  <Accordion title="Set up notifications">
+    Configure GitHub notifications or Slack integration to be alerted when auto-heal PRs are created.
+  </Accordion>
+
+  <Accordion title="Track heal rate">
+    Monitor how often tests need healing. High heal rates may indicate:
+    - Tests are too brittle
+    - Application is changing rapidly
+    - Element descriptions need improvement
+  </Accordion>
+</AccordionGroup>
+
+## Limitations
+
+Auto-healing works best for:
+- Element text changes
+- Layout and styling updates
+- Minor UI restructuring
+
+It may struggle with:
+- Major workflow changes
+- New features requiring new assertions
+- Complex multi-step interactions
+
+For significant changes, create new tests manually using the [TestDriver agent](/v7/copilot/creating-tests).

package/docs/v7/copilot/creating-tests.mdx

@@ -0,0 +1,156 @@
+---
+title: "Creating Tests"
+sidebarTitle: "Creating Tests"
+description: "Use the TestDriver agent and MCP to create tests through natural language"
+icon: "wand-magic-sparkles"
+---
+
+With GitHub Copilot and TestDriver's MCP server, you can create tests by chatting with an AI agent. The agent spawns a virtual machine, executes actions, and writes test code for you.
+
+## Start a Conversation
+
+Open GitHub Copilot Chat in VS Code. If your project has no other agents configured, the TestDriver agent is used by default. Otherwise, select **testdriver** from the agent dropdown in the chat panel.
+
+Describe what you want to test:
+
+```
+Create a test that logs into my app at https://myapp.com
+```
+
+The agent will:
+1. Start a new session and spawn a Linux virtual machine
+2. Launch Chrome and navigate to your URL
+3. Execute actions based on your instructions
+4. Write the test code to a `.test.mjs` file
+
+<Note>
+  The TestDriver agent appears in the agent selection dropdown if you have other agents configured (like `copilot-instructions.md` or other `.agent.md` files). Make sure to select **testdriver** to use TestDriver's MCP tools.
+</Note>
+
+## Live Preview Panel
+
+When the agent starts a session, a **live preview panel** opens in VS Code. This lets you:
+
+- **Watch tests execute** in real-time
+- **Interact with the VM** — click, type, and navigate manually
+- **Debug issues** — see exactly what the AI sees
+
+<Note>
+  The live preview panel requires the TestDriver VS Code extension. Set `TD_PREVIEW=ide` in your MCP configuration to enable it. See the [device configuration docs](/v7/device-config) for all preview options.
+</Note>
+
+## Interactive Workflow
+
+The recommended workflow for creating tests:
+
+<Steps>
+  <Step title="Describe Your Test">
+    Tell the agent what you want to test:
+
+    ```
+    Test the checkout flow on https://shop.example.com
+    ```
+  </Step>
+
+  <Step title="Watch and Guide">
+    The agent will start executing. Watch the preview panel and provide guidance if needed:
+
+    ```
+    The login modal appeared. Use test@example.com and password123
+    ```
+  </Step>
+
+  <Step title="Add Assertions">
+    Ask the agent to verify specific conditions:
+
+    ```
+    Assert that the order confirmation page is shown with the order number
+    ```
+  </Step>
+
+  <Step title="Review the Code">
+    The agent writes test code as it works. Review the generated `.test.mjs` file and make any adjustments.
+  </Step>
+</Steps>
+
+## Example Conversation
+
+Here's a complete example of creating a login test:
+
+```
+You: Create a test for the login flow at https://console.testdriver.ai
+
+Agent: I'll create a test for the login flow. Starting a session...
+       [Screenshot: Login page loaded]
+
+       I can see the login page. Let me find the email input.
+       [Screenshot: Email input highlighted]
+
+       Now I'll type the email and find the password field...
+
+You: Use test@example.com for the email
+
+Agent: Got it. Typing test@example.com...
+       [Screenshot: Email entered]
+
+       Now entering the password and clicking Sign In...
+       [Screenshot: Dashboard loaded]
+
+       ✅ Login successful! I've written the test to tests/login.test.mjs
+```
+
+## Generated Test Code
+
+The agent generates standard Vitest test files:
+
+```javascript tests/login.test.mjs
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+
+describe("Login Flow", () => {
+  it("should log in successfully", async (context) => {
+    const testdriver = TestDriver(context);
+
+    await testdriver.provision.chrome({
+      url: "https://console.testdriver.ai"
+    });
+
+    const emailInput = await testdriver.find("email input field");
+    await emailInput.click();
+    await testdriver.type("test@example.com");
+
+    const passwordInput = await testdriver.find("password input field");
+    await passwordInput.click();
+    await testdriver.type("password123");
+
+    const signInButton = await testdriver.find("Sign In button");
+    await signInButton.click();
+
+    const result = await testdriver.assert("dashboard is visible");
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+## Tips for Better Tests
+
+<AccordionGroup>
+  <Accordion title="Be specific with element descriptions">
+    Instead of "click the button", say "click the blue Sign In button in the header". More context helps the AI find the right element.
+  </Accordion>
+  <Accordion title="Add waits for dynamic content">
+    If your app has animations or loading states, tell the agent to wait:
+    ```
+    Wait for the loading spinner to disappear before continuing
+    ```
+  </Accordion>
+  <Accordion title="Use assertions liberally">
+    Add assertions after each major action to catch regressions early:
+    ```
+    Assert that the product was added to the cart
+    ```
+  </Accordion>
+  <Accordion title="Break complex flows into steps">
+    For long workflows, create the test incrementally and verify each step works before moving on.
+  </Accordion>
+</AccordionGroup>

package/docs/v7/copilot/github.mdx

@@ -0,0 +1,143 @@
+---
+title: "GitHub Integration"
+sidebarTitle: "GitHub"
+description: "Use TestDriver from GitHub's web interface and mobile app"
+icon: "github"
+---
+
+TestDriver works directly in GitHub's web interface and mobile app. The same MCP server that powers VS Code integration also works in GitHub, letting you create and manage tests from anywhere.
+
+## How It Works
+
+When you add a TestDriver agent file to your repository at `.github/agents/testdriver.agent.md`, GitHub Copilot can use TestDriver's MCP tools directly in:
+
+- GitHub.com (web browser)
+- GitHub Mobile app (iOS/Android)
+- Pull request conversations
+- Issue comments
+
+## Using TestDriver in GitHub Web
+
+<Steps>
+  <Step title="Navigate to Your Repository">
+    Open your repository on GitHub.com. Make sure you have the TestDriver agent file at `.github/agents/testdriver.agent.md`.
+  </Step>
+
+  <Step title="Start a Copilot Chat">
+    Click the **Copilot icon** in the GitHub interface to open a chat. You can find this in:
+    - The repository's Code tab
+    - Pull request pages
+    - Issue pages
+  </Step>
+
+  <Step title="Invoke the TestDriver Agent">
+    Start your message with `@testdriver`:
+
+    ```
+    @testdriver Create a test that verifies the homepage loads correctly at https://myapp.com
+    ```
+
+    The agent will spawn a sandbox environment and begin executing, just like in VS Code.
+  </Step>
+
+  <Step title="View Screenshots in Chat">
+    As the test runs, screenshots appear directly in the chat. You can see what the AI sees and provide guidance if needed.
+  </Step>
+</Steps>
+
+## Creating Tests from PR Comments
+
+You can create tests directly from pull request reviews. Comment on a PR and mention Copilot:
+
+```
+@copilot create a TestDriver test that verifies this new feature works.
+Test the checkout flow with a guest user.
+```
+
+Copilot will:
+1. Use the TestDriver MCP server
+2. Create a test based on your description
+3. Commit the test file to the PR branch
+
+This is useful for:
+- Adding test coverage during code review
+- Verifying bug fixes before merging
+- Creating regression tests for new features
+
+## Creating Tests from Issues
+
+You can also create tests from issue comments:
+
+```
+@copilot Use TestDriver to create a test that reproduces this bug.
+Navigate to /settings, change the theme, and verify it persists after refresh.
+```
+
+The test will be created in a new branch and linked to the issue.
+
+## Mobile App Support
+
+The GitHub Mobile app supports Copilot chat, which means you can use TestDriver from your phone:
+
+1. Open the GitHub app
+2. Navigate to your repository
+3. Tap the Copilot icon
+4. Type `@testdriver` followed by your request
+
+Screenshots and test progress appear in the chat, letting you create and debug tests on the go.
+
+## Example: PR Review Workflow
+
+Here's a complete workflow for adding tests during code review:
+
+<Steps>
+  <Step title="Developer Opens PR">
+    A developer opens a pull request with a new feature.
+  </Step>
+
+  <Step title="Reviewer Requests Tests">
+    The reviewer comments:
+    ```
+    @copilot Create a TestDriver test for this user registration flow.
+    Test both successful registration and validation errors.
+    ```
+  </Step>
+
+  <Step title="Copilot Creates Tests">
+    Copilot spawns TestDriver, creates the tests, and commits them to the PR branch.
+  </Step>
+
+  <Step title="Tests Run in CI">
+    The new tests run automatically in CI, validating the feature works as expected.
+  </Step>
+</Steps>
+
+## Agent File Reference
+
+The agent file at `.github/agents/testdriver.agent.md` contains the configuration for GitHub Copilot to use TestDriver. Here's the structure:
+
+```yaml
+---
+name: testdriver
+description: An expert at creating and refining automated tests using TestDriver.ai
+mcp-servers:
+  testdriver:
+    command: npx
+    args:
+      - -p
+      - testdriverai
+      - testdriverai-mcp
+    env:
+      TD_API_KEY: ${TD_API_KEY}
+---
+
+# TestDriver Expert
+
+You are an expert at writing automated tests using TestDriver...
+```
+
+The `TD_API_KEY` is pulled from your repository secrets when running in GitHub Actions or from your environment when using the web interface.
+
+<Warning>
+  Make sure `TD_API_KEY` is set in your repository secrets for CI workflows. Go to **Settings → Secrets and variables → Actions** to add it.
+</Warning>

package/docs/v7/copilot/running-tests.mdx

@@ -0,0 +1,149 @@
+---
+title: "Running Tests"
+sidebarTitle: "Running Tests"
+description: "Re-run tests with Vitest and use GUI mode for visual debugging"
+icon: "play"
+---
+
+After creating tests with the TestDriver agent, you can re-run them without starting a new MCP session. Tests are saved as standard Vitest files that run independently.
+
+## Running from VS Code (GUI Mode)
+
+For a visual testing experience, use the **Vitest extension**:
+
+<Steps>
+  <Step title="Open Test Explorer">
+    Click the **beaker icon** in the VS Code sidebar to open the Test Explorer. This shows all your test files and test cases.
+  </Step>
+
+  <Step title="Run Tests">
+    Click the **play button** next to any test file or individual test to run it. You can also:
+    - Run all tests with the "Run All" button
+    - Debug tests with the "Debug" button
+  </Step>
+
+  <Step title="View Results">
+    Test results appear inline:
+    - ✅ Green checkmark for passing tests
+    - ❌ Red X for failing tests
+    - Click on a failing test to see error details
+  </Step>
+</Steps>
+
+<Warning>
+  VS Code's Test Explorer only shows output for **failing tests**. To see output from passing tests (including screenshots and console logs), run tests from the terminal instead.
+</Warning>
+
+## Running from Terminal
+
+Use Vitest to run your tests from the command line to see full output:
+
+```bash
+# Run all tests
+vitest run
+
+# Run a specific test file
+vitest run tests/login.test.mjs
+
+# Run in watch mode (re-runs on file changes)
+vitest
+```
+
+See the [running tests guide](/v7/running-tests) for more CLI options and configuration.
+
+## Iterating on Tests
+
+When tests fail or need updates, you have two options:
+
+### Option 1: Ask the MCP Agent (Recommended)
+
+For discovering updated element descriptions or debugging failures, chat with the TestDriver agent:
+
+```
+The login test is failing because the form layout changed.
+Update tests/login.test.mjs to work with the new design.
+```
+
+The agent will:
+1. Start a new session
+2. Navigate to the page
+3. Analyze the current state
+4. Update the test code
+
+This is the best approach when:
+- Element text or layout has changed
+- You need to see what's currently on screen
+- The failure reason isn't obvious from the error message
+
+### Option 2: Edit the Code Directly
+
+For simple changes, edit the test files directly:
+
+```javascript
+// Change the element description
+const button = await testdriver.find("Submit Order button");
+
+// Add a wait
+await testdriver.wait(2000);
+
+// Update assertions
+const result = await testdriver.assert("order confirmation shows order ID");
+```
+
+This is faster for:
+- Updating text strings
+- Adjusting timeouts
+- Fixing typos
+
+## Test Configuration
+
+### Timeouts
+
+TestDriver tests require longer timeouts than typical unit tests. Your `vitest.config.mjs` should have:
+
+```javascript vitest.config.mjs
+import { defineConfig } from "vitest/config";
+
+export default defineConfig({
+  test: {
+    testTimeout: 900000,  // 15 minutes
+    hookTimeout: 900000,  // 15 minutes for setup/teardown
+  },
+});
+```
+
+### Environment Variables
+
+Tests use the `TD_API_KEY` environment variable. Set it in your `.env` file:
+
+```env .env
+TD_API_KEY=your-api-key-here
+```
+
+Or pass it when running tests:
+
+```bash
+TD_API_KEY=your-key vitest run
+```
+
+## Viewing Test Reports
+
+After each test run, TestDriver provides a link to the full test report:
+
+```
+TESTDRIVER_RUN_URL=https://console.testdriver.ai/runs/abc123
+```
+
+The report includes:
+- Video recording of the test
+- Screenshots at each step
+- Network logs and performance metrics
+- Console output and errors
+
+<Card
+  title="View Test Reports"
+  icon="chart-line"
+  href="https://console.testdriver.ai"
+>
+  Access all your test runs and recordings in the TestDriver console
+</Card>