@testdriverai/agent 7.11.229-canary → 7.11.232-test

package/docs/copilot/auto-healing.mdx

@@ -1,414 +1,130 @@
 ---
-title: "Prevent"
+title: "Prevent regressions"
 sidebarTitle: "Prevent"
-description: "Catch regressions automatically with CI, auto-healing, and GitHub integration"
+description: "Catch regressions before they merge, and fix tests when your app changes."
 icon: "shield-check"
 ---
 
-Prevent regressions by letting TestDriver run and repair itself on every change. Once you've [explored](/generating-tests), [taught it your app](/caching), [run it](/copilot/running-tests), [validated outcomes](/making-assertions), [adapted to your UI](/performing-actions), and learned to [debug with screenshots](/debugging-with-screenshots), the last step is closing the loop: wire TestDriver into your pull requests so regressions never reach production.
+Stop regressions before they reach your main branch.
 
-Your AI agent can run tests on every PR, investigate failures, and propose fixes — and you can drive all of it from GitHub itself, whether you're at your desk or on your phone.
+- Run your tests on every pull request.
+- Let the TestDriver agent review each change.
+- Ask the agent to fix a test when your app changes.
 
-## Use TestDriver in GitHub
+TestDriver tests don't use selectors, so many UI changes never break them.
 
-TestDriver works directly in GitHub's web interface and mobile app. The same MCP server that powers VS Code integration also works in GitHub, letting you create and manage tests from anywhere.
+## Tests that adapt to UI changes
 
-### How It Works
+A TestDriver test describes elements the way a person would, such as "the Sign in button in the header". It finds them by looking at the screen.
 
-When you add a TestDriver agent file to your repository at `.github/agents/testdriver.agent.md`, GitHub Copilot can use TestDriver's MCP tools directly in:
+These changes don't break the test, because the description still matches what's on screen:
 
-- GitHub.com (web browser)
-- GitHub Mobile app (iOS/Android)
-- Pull request conversations
-- Issue comments
+- Restyling a button.
+- Moving it.
+- Rewriting the markup around it.
 
-### Using TestDriver in GitHub Web
+A selector-based test would fail on the same change.
 
-<Steps>
-  <Step title="Navigate to Your Repository">
-    Open your repository on GitHub.com. Make sure you have the TestDriver agent file at `.github/agents/testdriver.agent.md`.
-  </Step>
+When the screen does change, the [cache](/caching) notices that it no longer matches the last passing run. It asks the AI to find the element again and learns its new location for the next run.
 
-  <Step title="Start a Copilot Chat">
-    Click the **Copilot icon** in the GitHub interface to open a chat. You can find this in:
-    - The repository's Code tab
-    - Pull request pages
-    - Issue pages
-  </Step>
+Descriptions that say what an element is and where it sits survive change best:
 
-  <Step title="Invoke the TestDriver Agent">
-    Start your message with `@testdriver`:
+```javascript
+// Survives restyling and layout changes
+await testdriver.find('primary Sign up button in the hero section').click();
 
-    ```
-    @testdriver Create a test that verifies the homepage loads correctly at https://myapp.com
-    ```
-
-    The agent will spawn a sandbox environment and begin executing, just like in VS Code.
-  </Step>
-
-  <Step title="View Screenshots in Chat">
-    As the test runs, screenshots appear directly in the chat. You can see what the AI sees and provide guidance if needed.
-  </Step>
-</Steps>
-
-### Creating Tests from PR Comments
-
-You can create tests directly from pull request reviews. Comment on a PR and mention Copilot:
-
-```
-@copilot create a TestDriver test that verifies this new feature works.
-Test the checkout flow with a guest user.
+// Too vague: may match the wrong button after a redesign
+await testdriver.find('button').click();
 ```
 
-Copilot will:
-1. Use the TestDriver MCP server
-2. Create a test based on your description
-3. Commit the test file to the PR branch
+## Run tests on every pull request
 
-This is useful for:
-- Adding test coverage during code review
-- Verifying bug fixes before merging
-- Creating regression tests for new features
+Run your suite in GitHub Actions, so a change that breaks a flow fails its pull request.
 
-### Creating Tests from Issues
+The `testdriverai/action` step authenticates with GitHub OIDC, so there's no API key secret to store. `npx testdriverai init` writes a workflow like this for you:
 
-You can also create tests from issue comments:
+```yaml .github/workflows/testdriver.yml
+name: TestDriver.ai Tests
 
-```
-@copilot Use TestDriver to create a test that reproduces this bug.
-Navigate to /settings, change the theme, and verify it persists after refresh.
-```
-
-The test will be created in a new branch and linked to the issue.
-
-### Mobile App Support
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
 
-The GitHub Mobile app supports Copilot chat, which means you can use TestDriver from your phone:
-
-1. Open the GitHub app
-2. Navigate to your repository
-3. Tap the Copilot icon
-4. Type `@testdriver` followed by your request
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write   # lets the job mint the OIDC token TestDriver accepts
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: 'npm'
+      - run: npm ci
+      - name: Authenticate to TestDriver
+        uses: testdriverai/action@stable
+      - name: Run TestDriver tests
+        run: npx vitest run
+```
 
-Screenshots and test progress appear in the chat, letting you create and debug tests on the go.
+OIDC needs the TestDriver GitHub app authorized for your organization.
 
-### Example: PR Review Workflow
+For other CI providers and runner setups, see [Run tests in CI/CD](/ci-cd).
 
-Here's a complete workflow for adding tests during code review:
+## Review every pull request
 
-<Steps>
-  <Step title="Developer Opens PR">
-    A developer opens a pull request with a new feature.
-  </Step>
+With the [TestDriver GitHub app](/quickstart-github) installed, the TestDriver agent reviews each pull request when it's opened and after every push.
 
-  <Step title="Reviewer Requests Tests">
-    The reviewer comments:
-    ```
-    @copilot Create a TestDriver test for this user registration flow.
-    Test both successful registration and validation errors.
-    ```
-  </Step>
+- It reads the change for bugs.
+- When the change is something a user would see, it runs it in a sandbox to prove what it finds.
 
-  <Step title="Copilot Creates Tests">
-    Copilot spawns TestDriver, creates the tests, and commits them to the PR branch.
-  </Step>
+This catches regressions that your existing tests don't cover. See [the TestDriver agent](/agent) for what a review contains.
 
-  <Step title="Tests Run in CI">
-    The new tests run automatically in CI, validating the feature works as expected.
-  </Step>
-</Steps>
+## Fix a test when your app changes
 
-### Agent File Reference
+Sometimes a test fails because the app changed on purpose, such as a renamed button or a reworked flow. Ask the TestDriver agent to update the test.
 
-The agent file at `.github/agents/testdriver.agent.md` contains the configuration for GitHub Copilot to use TestDriver. Here's the structure:
+On GitHub, mention it on the pull request or an issue:
 
-```yaml
----
-name: testdriver
-description: An expert at creating and refining automated tests using TestDriver.ai
-mcp-servers:
-  testdriver:
-    command: npx
-    args:
-      - -p
-      - testdriverai
-      - testdriverai-mcp
-    env:
-      TD_API_KEY: ${TD_API_KEY}
----
+```text
+@testdriverai The checkout test fails after this redesign because the Submit button is now called Place order. Update the test.
+```
 
-# TestDriver Expert
+The agent then:
 
-You are an expert at writing automated tests using TestDriver...
-```
+1. Reruns the flow in a sandbox and sees the current screen.
+2. Updates the test.
+3. Opens a pull request with the fix for you to review.
 
-The `TD_API_KEY` is pulled from your repository secrets when running in GitHub Actions or from your environment when using the web interface.
+You can make the same request from your editor through the [Claude Code plugin](/quickstart-claude) or the [VS Code extension](/quickstart-vscode).
 
 <Warning>
-  Make sure `TD_API_KEY` is set in your repository secrets for CI workflows. Go to **Settings → Secrets and variables → Actions** to add it.
+  Review the agent's fix before you merge it. A test updated to pass may no longer check what you meant. Confirm that the assertions still describe the behavior you want.
 </Warning>
 
-## Auto-Healing Tests
-
-Auto-healing tests automatically fix themselves when your application changes. By integrating an AI coding agent with TestDriver in your CI pipeline, you can have AI investigate test failures and propose fixes. The example below uses GitHub Copilot in GitHub Actions, but the same approach works with any AI agent that can run TestDriver's MCP server.
-
-### How It Works
-
-When a test fails in CI:
-
-1. **GitHub Actions detects the failure**
-2. **The AI agent spawns TestDriver** with access to the MCP server
-3. **The agent investigates** by running the failing test and analyzing what changed
-4. **The agent creates a fix** by updating the test code
-5. **A pull request is opened** with the proposed changes for review
-
-### Setting Up Auto-Healing
-
-<Steps>
-  <Step title="Add Repository Secrets">
-    Add your TestDriver API key to your repository secrets:
-
-    1. Go to **Settings → Secrets and variables → Actions**
-    2. Click **New repository secret**
-    3. Add `TD_API_KEY` with your API key value
-  </Step>
-
-  <Step title="Create the Auto-Heal Workflow">
-    Add a GitHub Actions workflow that triggers on test failures:
-
-    ```yaml .github/workflows/auto-heal.yml
-    name: Auto-Heal Tests
-
-    on:
-      workflow_run:
-        workflows: ["Tests"]  # Your main test workflow
-        types: [completed]
-        branches: [main]
-
-    jobs:
-      auto-heal:
-        if: ${{ github.event.workflow_run.conclusion == 'failure' }}
-        runs-on: ubuntu-latest
-        permissions:
-          contents: write
-          pull-requests: write
-
-        steps:
-          - uses: actions/checkout@v4
-
-          - name: Setup Node.js
-            uses: actions/setup-node@v4
-            with:
-              node-version: "20"
-
-          - name: Install dependencies
-            run: npm ci
-
-          - name: Run failing tests and capture output
-            id: tests
-            continue-on-error: true
-            run: |
-              vitest run 2>&1 | tee test-output.txt
-              echo "output<<EOF" >> $GITHUB_OUTPUT
-              cat test-output.txt >> $GITHUB_OUTPUT
-              echo "EOF" >> $GITHUB_OUTPUT
-            env:
-              TD_API_KEY: ${{ secrets.TD_API_KEY }}
-
-          - name: Invoke Copilot to fix tests
-            uses: github/copilot-action@v1
-            with:
-              prompt: |
-                @testdriver The following tests failed:
-
-                ${{ steps.tests.outputs.output }}
-
-                Please investigate each failure by:
-                1. Running the failing test
-                2. Analyzing what changed in the UI or behavior
-                3. Updating the test code to fix the issue
-
-                Create a commit with your changes.
-            env:
-              TD_API_KEY: ${{ secrets.TD_API_KEY }}
-
-          - name: Create Pull Request
-            uses: peter-evans/create-pull-request@v5
-            with:
-              title: "Auto-heal: Fix failing tests"
-              body: |
-                This PR was automatically generated by the auto-heal workflow.
-
-                ## Changes
-                The following tests were updated to fix failures detected in CI.
-
-                ## Review
-                Please review the changes carefully before merging.
-              branch: auto-heal/${{ github.run_id }}
-              commit-message: "fix: auto-heal failing tests"
-    ```
-  </Step>
-
-  <Step title="Configure Test Workflow">
-    Make sure your main test workflow has a name that matches the `workflows` trigger:
-
-    ```yaml .github/workflows/tests.yml
-    name: Tests  # Must match the workflow_run trigger
-
-    on:
-      push:
-        branches: [main]
-      pull_request:
-
-    jobs:
-      test:
-        runs-on: ubuntu-latest
-        steps:
-          - uses: actions/checkout@v4
-          - uses: actions/setup-node@v4
-            with:
-              node-version: "20"
-          - run: npm ci
-          - run: vitest run
-            env:
-              TD_API_KEY: ${{ secrets.TD_API_KEY }}
-    ```
-  </Step>
-</Steps>
-
-### Example: Button Text Change
-
-Here's what auto-healing looks like when a button's text changes:
-
-<Steps>
-  <Step title="Application Changes">
-    A developer changes a button's text from "Submit" to "Send":
-
-    ```html
-    <!-- Before -->
-    <button>Submit</button>
-
-    <!-- After -->
-    <button>Send</button>
-    ```
-  </Step>
-
-  <Step title="Test Fails">
-    The test fails because it's looking for the old text:
-
-    ```javascript
-    // This now fails
-    const submitButton = await testdriver.find("Submit button");
-    ```
-  </Step>
-
-  <Step title="Auto-Heal Triggers">
-    The auto-heal workflow runs, and the AI agent investigates:
-
-    ```
-    The test is looking for a "Submit button" but I see a "Send button"
-    on the page. The button functionality is the same, just the text changed.
-    I'll update the test to use the new text.
-    ```
-  </Step>
-
-  <Step title="PR Created">
-    A pull request is opened with the fix:
-
-    ```javascript
-    // Updated by auto-heal
-    const submitButton = await testdriver.find("Send button");
-    ```
-  </Step>
-</Steps>
-
-### Configuration Options
-
-#### Selective Auto-Healing
-
-You can limit auto-healing to specific test files or patterns:
-
-```yaml
-- name: Run failing tests
-  run: |
-    # Only heal tests in the e2e directory
-    vitest run tests/e2e/ 2>&1 | tee test-output.txt
-```
+## When updating a test isn't enough
 
-#### Manual Approval
+Updating a description or an assertion works well when an element is renamed, restyled, or moved.
 
-For safety, you can require manual approval before auto-heal runs:
+When a flow changes substantially, such as a new step in checkout or a redesigned form, describe the new flow. Have the agent [generate a fresh test](/generating-tests) instead.
 
-```yaml
-jobs:
-  auto-heal:
-    environment: auto-heal  # Requires approval
-```
-
-Configure the environment in **Settings → Environments** with required reviewers.
-
-#### Limiting Changes
-
-Add instructions to constrain what the AI can change:
-
-```yaml
-- name: Invoke Copilot to fix tests
-  uses: github/copilot-action@v1
-  with:
-    prompt: |
-      @testdriver Fix the failing tests.
-
-      Rules:
-      - Only update element selectors and text matching
-      - Do not change test logic or assertions
-      - Do not add or remove tests
-      - Keep changes minimal
-```
+## Next steps
 
-### Best Practices
-
-<AccordionGroup>
-  <Accordion title="Always review auto-heal PRs">
-    Auto-heal is a tool, not a replacement for human judgment. Review all changes before merging to ensure the test still validates what you intended.
-  </Accordion>
-
-  <Accordion title="Use descriptive element descriptions">
-    Tests with clear, semantic descriptions are easier for the AI to heal:
-    ```javascript
-    // ✅ Good - describes purpose
-    await testdriver.find("primary call-to-action button in the hero section");
-
-    // ❌ Bad - too vague
-    await testdriver.find("button");
-    ```
-  </Accordion>
-
-  <Accordion title="Set up notifications">
-    Configure GitHub notifications or Slack integration to be alerted when auto-heal PRs are created.
-  </Accordion>
-
-  <Accordion title="Track heal rate">
-    Monitor how often tests need healing. High heal rates may indicate:
-    - Tests are too brittle
-    - Application is changing rapidly
-    - Element descriptions need improvement
-  </Accordion>
-</AccordionGroup>
-
-### Limitations
-
-Auto-healing works best for:
-- Element text changes
-- Layout and styling updates
-- Minor UI restructuring
-
-It may struggle with:
-- Major workflow changes
-- New features requiring new assertions
-- Complex multi-step interactions
-
-For significant changes, create new tests by going back to [Explore](/generating-tests) and generating them with the TestDriver agent.
-
-<Card title="View Your Runs" icon="chart-line" href="https://console.testdriver.ai">
-  Open the TestDriver console to monitor test runs, healing PRs, and CI results across your projects.
-</Card>
+<CardGroup cols={2}>
+  <Card title="Run tests in CI/CD" icon="code-branch" href="/ci-cd">
+    Set up GitHub Actions and other CI providers.
+  </Card>
+  <Card title="The TestDriver agent" icon="hand-horns" href="/agent">
+    How the agent reviews pull requests and fixes tests.
+  </Card>
+  <Card title="Locate elements" icon="magnifying-glass" href="/locating-elements">
+    Write descriptions that keep working as your UI changes.
+  </Card>
+  <Card title="Debug a failing test" icon="bug" href="/debugging-with-screenshots">
+    See what the screen showed when a test failed.
+  </Card>
+</CardGroup>