playwright-e2e-mcp 0.3.2 → 0.4.0

package/skill/playwright-e2e-mcp/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: playwright-e2e-mcp
-description: Runs, debugs and fixes Playwright end-to-end tests through the playwright-e2e-mcp server, using structured run results, trace-based failure diagnostics, verified locators, self-healing fixes that re-run and revert if still red, per-test flaky verdicts that separate parallelism bugs from races, signed-in page inspection and local run history. Use when the user asks to run Playwright tests, fix a failing or broken e2e test, find out why a test is flaky or which tests are unstable, repair a selector that no longer matches, write a new Playwright test, or check a page for visual regressions.
+description: Runs, debugs and fixes Playwright end-to-end tests through the playwright-e2e-mcp server, using structured run results grouped by root cause, trace-based failure diagnostics, verified locators, self-healing fixes that re-run and revert if still red, per-test flaky verdicts that separate parallelism, test-order and timing bugs, signed-in page inspection, WCAG accessibility audits and local run history, plus a guide to writing tests that stay green. Use when the user asks to run Playwright tests, fix a failing or broken e2e test, find out why a test is flaky or which tests are unstable, repair a selector that no longer matches, write or review a Playwright test, check a page for accessibility or visual regressions.
 ---
 
 # Playwright E2E with playwright-e2e-mcp
@@ -13,22 +13,26 @@ and concrete next steps. It can also propose and apply a verified fix for a brok
 locator, inspect pages behind a login, run long suites in the background and keep
 a local history of every run.
 
-This skill is written for playwright-e2e-mcp 0.3.0 and works with 0.2.0 or
+This skill is written for playwright-e2e-mcp 0.4.0 and works with 0.2.0 or
 newer. If `suggest-fix`, `get-run-status` or `analyze-history` is missing, the
 server is older than 0.2.0: use the steps that do not need those tools and tell
 the user to update it. `diagnose-flaky`'s per-test table and `isolate` option
-arrived in 0.3.0.
+arrived in 0.3.0; `audit-accessibility`, `diagnose-flaky`'s `order` option and
+root-cause grouping in `run-test` arrived in 0.4.0.
 
 ## Before you start
 
 1. Check that the server's tools are available. They are named `run-test`,
    `get-run-status`, `get-failure`, `suggest-fix`, `inspect-page`,
    `validate-selector`, `list-tests`, `generate-e2e-test`,
-   `compare-visual-state`, `diagnose-flaky` and `analyze-history` (your client
+   `compare-visual-state`, `audit-accessibility`, `diagnose-flaky` and
+   `analyze-history` (your client
    may prefix them, e.g. `mcp__playwright-e2e__run-test`).
 2. If they are all missing, stop and tell the user the MCP server is not
    connected. Point them to `references/setup.md` for the one-line install for
-   their client. Do not fall back to shelling out to `npx playwright test` unless
+   their client (`npx -y playwright-e2e-mcp init` sets up every client the
+   project uses). Writing or reviewing tests by hand still works: follow
+   `references/writing-tests.md`. Do not fall back to shelling out to `npx playwright test` unless
    the user asks. If only some are missing, the server may run a smaller tool
    profile (`--tools=core` exposes `run-test`, `get-run-status`, `get-failure`,
    `suggest-fix`, `diagnose-flaky` and `list-tests`); work with what is there and
@@ -49,7 +53,10 @@ Follow these steps in order. Do not edit code before step 3.
    Omit `testFiles` only when the user asks for the whole suite. Failing tests are
    retried once by default, so a test that passes on retry is reported as
    **flaky**, not failed. For a long suite pass `background: true` and poll
-   `get-run-status` with `waitSeconds: 30`.
+   `get-run-status` with `waitSeconds: 30`. When several tests fail the same
+   way, the result says how many distinct causes there are and shows each
+   cause once: fix the cause, not each test (a dead server or one changed
+   shared component often fails dozens of tests).
 2. **Read the failure.** For each failure, call `get-failure` with its 1-based
    `index`. Read the message and code frame, expected vs actual, the failure kind,
    the DOM snapshot at failure, the failed action and its selector, failed network
@@ -138,6 +145,20 @@ failing ones with one worker, and reads each one:
 - **flaky even alone**: the race is inside the test or the app.
 - **fails alone too**: a real bug, not parallelism.
 
+When a test fails in the full suite but `diagnose-flaky` on that test alone says
+**NOT REPRODUCING**, suspect test order: call it again with `order: true` and
+the spec file(s) in `testFiles` (Playwright 1.64+). It runs the files in random
+orders with one worker, replays a failing order, and reads each test:
+
+- **order dependent**: fails after certain tests and fails again on replay. The
+  result names the test it **relies on** (state it needs another test to create)
+  or is **broken by** (state another test leaves behind), and the
+  `--shuffle=<seed>` command that reproduces it. Fix it by giving the test its
+  own setup (`beforeEach`, a fixture, its own data) and cleaning up in the test
+  that leaks; do not pin the file to serial mode.
+- **flaky regardless of order**: the failing order passed on replay: timing.
+- **fails in every order**: a real bug.
+
 Pass `separateProcesses: true` only when the flake seems tied to global setup or
 a cold start, since one process per run is slower. `get-failure` works on the
 last failing diagnose run straight after, and each repetition is recorded in the
@@ -158,9 +179,20 @@ pass/fail timeline.
   file it writes, then `run-test` it. Pass `write: false` to preview without
   writing.
 - **Set up CI:** when the user wants the same diagnosis on every pull request,
-  point them to the GitHub Action (`uses: xsignon-secure/E2E@v0.3.0`) or the
+  point them to the GitHub Action (`uses: xsignon-secure/E2E@v0.4.0`) or the
   `npx -y playwright-e2e-mcp ci` command in `references/setup.md`. It writes the
   run, flaky verdicts and fix diffs to the job summary and one PR comment.
+- **Accessibility:** `audit-accessibility` with `url` runs axe-core against
+  WCAG 2.2 A/AA (`standard` picks another level or `best-practice`). It takes
+  the same `storageState`, `headers` and `actions` as `inspect-page`, so audit
+  the signed-in pages and open dialogs, not only the landing page. Each
+  violation lists the offending elements as verified locators and the fix; fix
+  the worst (`critical`, `serious`) first, re-run the audit, then add the
+  `AxeBuilder` check it prints to the spec that reaches that page so it stays
+  fixed. Pass `minImpact: "serious"` for a short list.
+- **Write or review a test:** follow `references/writing-tests.md` (locators,
+  web-first assertions, auth setup, test data, network mocking, flake
+  patterns), with or without the server.
 - **Visual regression:** `compare-visual-state` with `url` and a `name`. The first
   call saves a baseline under `.pw-mcp/visual/`; later calls report changed
   regions and color shifts and write a diff image. Use `action: "baseline"` only
@@ -194,3 +226,5 @@ When you finish, tell the user, in this order:
 - `references/failure-kinds.md`: failure kinds and error codes, and what to do.
 - `references/example-session.md`: a worked fix-a-failing-test session with
   `suggest-fix`.
+- `references/writing-tests.md`: how to write Playwright tests that stay green,
+  and the review checklist.

package/skill/playwright-e2e-mcp/references/setup.md

@@ -5,10 +5,11 @@ This skill only gives the agent instructions. The tools come from the
 
 ## Requirements
 
-- playwright-e2e-mcp **0.2.0 or newer**, 0.3.0 recommended. `suggest-fix`,
+- playwright-e2e-mcp **0.2.0 or newer**, 0.4.0 recommended. `suggest-fix`,
   `get-run-status`, `analyze-history`, verified locators and signed-in page
   inspection arrived in 0.2.0; per-test flaky verdicts, `isolate`, tool profiles
-  and the `ci` command in 0.3.0. Check with
+  and the `ci` command in 0.3.0; `audit-accessibility`, the `order` check
+  (needs Playwright 1.64+) and root-cause grouping in 0.4.0. Check with
   `npx -y playwright-e2e-mcp --version`.
 
 - Node.js 20 or newer.
@@ -80,6 +81,7 @@ Set these in the server's `env` block. None of them are secrets except
 | `PW_MCP_ALLOWED_ROOTS` | none | Extra directories allowed as `projectRoot` (`:`-separated, `;` on Windows) |
 | `PW_MCP_BLOCK_PRIVATE_URLS` | off for stdio | `1` makes the URL tools refuse loopback, private-network and cloud-metadata addresses |
 | `PW_MCP_STORAGE_STATE` | none | Default Playwright storageState JSON (inside the project) for the page tools, so signed-in pages open logged in |
+| `PW_MCP_SHUFFLE_SEEDS` | random | Seeds for `diagnose-flaky`'s `order` pass (comma separated), to replay reported orders |
 | `PW_MCP_HISTORY` | on | `0` stops recording runs to `<project>/.playwright-e2e-mcp/history.jsonl` (this turns off `analyze-history`) |
 | `LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` or `silent` |
 | `PW_MCP_HTTP_TOKEN` | none | HTTP bridge only: bearer token that unlocks all tools |
@@ -110,7 +112,7 @@ Exit code 0 means passed, 1 tests failed, 2 the run itself failed. On GitHub
 Actions it also writes to the job summary. As an Action:
 
 ```yaml
-- uses: xsignon-secure/E2E@v0.3.0
+- uses: xsignon-secure/E2E@v0.4.0
   with:
     flaky-runs: 3
     suggest: 2

package/skill/playwright-e2e-mcp/references/tools.md

@@ -24,6 +24,9 @@ Runs Playwright and returns stats, failures, failure kinds and fix hints.
 
 Traces are captured on failure automatically so `get-failure` can show the DOM.
 Clients that send a progress token get live progress while the run goes.
+Failures that share a kind and first error line are grouped (0.4.0+): each
+cause is shown once in full, the other tests under it as one line each, and
+their `index` numbers stay the ones `get-failure` takes.
 
 ## get-failure
 
@@ -113,6 +116,7 @@ Baselines live in `.pw-mcp/visual/`.
 | `testFiles` | string[] | Default: tests that failed in the last run |
 | `runs` | number | 2 to 10 (default 3) |
 | `isolate` | boolean | Repeat fully parallel (`workers` default 4), then repeat the failing tests with one worker (0.3.0+) |
+| `order` | boolean | Run the target files in `runs` random orders with one worker (`--shuffle`, Playwright 1.64+), replay a failing order, and name the test it relies on or is broken by (0.4.0+) |
 | `separateProcesses` | boolean | One Playwright process per run instead of one `--repeat-each` process; slower (0.3.0+) |
 | `browser`, `headed`, `workers`, `config` | | As `run-test` |
 | `timeoutMs` | number | Per run (default 120000); the single `--repeat-each` process gets this × `runs` |
@@ -121,9 +125,30 @@ Runs with retries disabled and returns, per test, passes, failures, duration
 spread, first error and `CONSISTENTLY FAILING`, `FLAKY` or `NOT REPRODUCING`,
 plus the verdict block for the worst test. With `isolate`, a second table reads
 each failing test as "fails only in parallel", "flaky even alone" or "fails
-alone too". Versions before 0.3.0 return one run-level verdict and a per-run
+alone too". With `order`, a third section reads each test as "order
+dependent", "flaky regardless of order" or "fails in every order", with the
+`--shuffle=<seed>` command that reproduces it. `PW_MCP_SHUFFLE_SEEDS=1,2,3`
+on the server replays given seeds instead of random ones. Versions before 0.3.0 return one run-level verdict and a per-run
 table instead.
 
+## audit-accessibility
+
+Runs axe-core (0.4.0+) on a live page and returns WCAG violations, worst first.
+
+| Argument | Type | Notes |
+| --- | --- | --- |
+| `url` | string | Required |
+| `standard` | `wcag2a` \| `wcag2aa` \| `wcag21aa` \| `wcag22aa` \| `best-practice` \| `all` | Default `wcag22aa` (WCAG 2.0 to 2.2, A and AA) |
+| `minImpact` | `critical` \| `serious` \| `moderate` \| `minor` | Hide violations below this impact |
+| `selector` | string | Audit only this part of the page |
+| `storageState`, `headers`, `actions`, `viewport` | | As `inspect-page` |
+| `waitFor`, `waitUntil`, `timeoutMs` | | `waitUntil` default `load`; `timeoutMs` default 60000 |
+
+Each violation has its rule id, impact, WCAG criteria, a help link, up to three
+offending elements as verified locators with what to change, and the count of
+the rest. The result ends with an `@axe-core/playwright` check to paste into
+the spec.
+
 ## suggest-fix
 
 | Argument | Type | Notes |

package/skill/playwright-e2e-mcp/references/writing-tests.md

@@ -0,0 +1,170 @@
+# Writing Playwright tests that stay green
+
+Use this when you write a new test, extend one, or review one. It applies with or
+without the MCP server; where a tool helps, it is named.
+
+## 1. Locators
+
+Pick the first that works, and check it with `validate-selector` (exactly 1 match)
+or take it from `inspect-page` with `view: "locators"`:
+
+1. `page.getByRole('button', { name: 'Place order' })`: what a user and a screen
+   reader see. Add `exact: true` when the name is a prefix of another.
+2. `page.getByLabel('Email')`, `page.getByPlaceholder('Search')` for form fields.
+3. `page.getByText('Order confirmed')` for non-interactive content.
+4. `page.getByTestId('checkout')` when the app has test ids (set a custom
+   attribute with `testIdAttribute` in the config).
+5. A short CSS selector on a stable attribute, as a last resort.
+
+Narrow instead of indexing: `page.getByRole('listitem').filter({ hasText: 'Milk' })`,
+`row.getByRole('button', { name: 'Delete' })`. Avoid `nth()`, `:nth-child`,
+XPath, generated class names (`.css-1x2y3z`) and long descendant chains. Locators
+are strict: an action on a locator that matches several elements throws, which
+is a feature. Fix the locator rather than adding `.first()`.
+
+## 2. Waiting and assertions
+
+- Use web-first assertions, which retry until the timeout:
+  `await expect(page.getByRole('alert')).toHaveText('Saved')`, `toBeVisible()`,
+  `toHaveURL(/\/orders\/\d+/)`, `toHaveCount(3)`, `toBeEnabled()`.
+- Never `expect(await locator.isVisible()).toBe(true)` or
+  `expect(await locator.textContent())`: they read once and do not wait.
+- Never `page.waitForTimeout(…)`. Wait for the thing you need: an assertion, a
+  response, or a URL.
+- Actions already wait for the element to be visible, enabled and stable; do
+  not add `waitForSelector` before `click`.
+- For a value that is not in the DOM, poll: `await expect.poll(() => api.count()).toBe(3)`.
+- Wait for a request triggered by an action by starting the wait first:
+
+  ```ts
+  const saved = page.waitForResponse((r) => r.url().includes('/api/orders') && r.ok());
+  await page.getByRole('button', { name: 'Save' }).click();
+  await saved;
+  ```
+- `waitForLoadState('networkidle')` is discouraged: apps with polling or
+  analytics never go idle. Assert on the content you need instead.
+- Use `expect.soft` only when later checks still make sense after a failure.
+
+## 3. Signing in
+
+Log in once, not in every test. Use a setup project that saves the session and
+make the other projects depend on it:
+
+```ts
+// tests/auth.setup.ts
+import { test as setup, expect } from '@playwright/test';
+setup('sign in', async ({ page }) => {
+  await page.goto('/login');
+  await page.getByLabel('Email').fill(process.env.E2E_USER!);
+  await page.getByLabel('Password').fill(process.env.E2E_PASSWORD!);
+  await page.getByRole('button', { name: 'Sign in' }).click();
+  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
+  await page.context().storageState({ path: 'playwright/.auth/user.json' });
+});
+```
+
+```ts
+// playwright.config.ts
+projects: [
+  { name: 'setup', testMatch: /.*\.setup\.ts/ },
+  { name: 'chromium', use: { ...devices['Desktop Chrome'], storageState: 'playwright/.auth/user.json' }, dependencies: ['setup'] },
+],
+```
+
+Keep `playwright/.auth/` in `.gitignore`. Use one file per role (admin, member).
+Credentials come from environment variables, never from the spec. Pass the same
+file as `storageState` to `inspect-page`, `validate-selector` and
+`audit-accessibility` to look at signed-in pages.
+
+## 4. Test data and isolation
+
+Every test must pass alone, in any order, and in parallel:
+
+- Create the data a test needs in the test or a fixture, ideally through the API
+  (`request.post('/api/products', { data })`), not through the UI and not by
+  relying on a test that ran earlier.
+- Make it unique per worker and run:
+  `` `user-${testInfo.workerIndex}-${Date.now()}@example.com` ``.
+- Clean up what you created, or create it in a namespace that is reset.
+- Do not share mutable module or `globalThis` state between tests.
+
+Check it: `diagnose-flaky` with `isolate: true` finds tests that only fail in
+parallel; `order: true` finds tests that depend on another test running first.
+
+## 5. Network and time
+
+- Mock third parties (payments, maps, analytics) with `page.route`, and mock your
+  own API only for states that are hard to reach (errors, empty lists):
+
+  ```ts
+  await page.route('**/api/orders', (route) => route.fulfill({ status: 500, json: { error: 'down' } }));
+  ```
+- Replay a recorded session with `page.routeFromHAR` when the backend is slow or
+  unstable.
+- Control time with `page.clock` (`install`, `setFixedTime`, `fastForward`)
+  instead of waiting for timers, and pin `timezoneId` and `locale` in the config
+  when dates are on screen.
+
+## 6. Structure
+
+- One user journey per test, named for the behavior: `places an order with a saved card`.
+- Group steps with `await test.step('pay', async () => { … })` so traces and
+  reports read like the journey.
+- Prefer fixtures for setup and shared helpers; use a page object only for a
+  page several specs drive. Keep assertions in the test, not in page objects.
+- Tag suites (`test('…', { tag: '@smoke' }, …)`) and run them with `--grep @smoke`.
+
+## 7. Config that prevents trouble
+
+```ts
+export default defineConfig({
+  fullyParallel: true,
+  forbidOnly: !!process.env.CI,
+  retries: process.env.CI ? 2 : 0,
+  use: { baseURL: 'http://localhost:3000', trace: 'on-first-retry' },
+  webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI },
+});
+```
+
+`webServer` starts the app so runs never fail with "connection refused". Traces
+on retry are what `get-failure` reads.
+
+## 8. Flake patterns
+
+| Symptom | Usual cause | Fix |
+| --- | --- | --- |
+| Passes locally, times out in CI | Fixed sleeps, slow CI machine, no `webServer` | Web-first assertions; `webServer`; wait for the response |
+| "strict mode violation" | Locator matches several elements | Filter or scope the locator |
+| Element "detached from the DOM" | List re-renders after data loads | Assert the loaded state first, then act |
+| Fails only with other tests | Shared user or data | Unique data per test or worker (`isolate: true` confirms) |
+| Fails only after certain tests | Order dependence | Own setup per test (`order: true` names the test) |
+| Wrong date or number format | Time zone or locale | Set `timezoneId`, `locale`, `page.clock` |
+| Click does nothing | Overlay, animation, or element not ready | Assert the overlay is gone; assert enabled |
+
+Never fix flakiness with retries, longer timeouts or `waitForTimeout` alone.
+
+## 9. Accessibility and structure checks
+
+- Run `audit-accessibility` on new pages, then keep them clean with
+  `@axe-core/playwright`:
+
+  ```ts
+  import AxeBuilder from '@axe-core/playwright';
+  const { violations } = await new AxeBuilder({ page }).withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa']).analyze();
+  expect(violations.map((v) => v.id)).toEqual([]);
+  ```
+- Lock the structure of a component with
+  `await expect(page.getByRole('navigation')).toMatchAriaSnapshot(…)`.
+- For visual checks, `compare-visual-state` locally or `toHaveScreenshot()` in
+  the suite, with animations disabled and dynamic content masked.
+
+## 10. Review checklist
+
+- [ ] Every locator is role, label, text or test id based, and unique.
+- [ ] No `waitForTimeout`, no assertions on one-off reads.
+- [ ] The test creates its own data and passes alone and in any order.
+- [ ] Sign-in comes from a saved session, credentials from the environment.
+- [ ] Third-party calls are mocked; time-dependent output is pinned.
+- [ ] The test name describes the behavior; steps are grouped.
+- [ ] It ran green with `run-test`, and three times in a row with
+      `diagnose-flaky` (`runs: 3`) before you call it done.