playwright-e2e-mcp 0.4.2 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (56) hide show
  1. package/README.md +238 -924
  2. package/dist/http.d.ts +2 -1
  3. package/dist/http.js +2 -1
  4. package/dist/http.js.map +1 -1
  5. package/dist/server.d.ts +2 -2
  6. package/dist/server.js +31 -4
  7. package/dist/server.js.map +1 -1
  8. package/dist/tools/analyze-history.d.ts +12 -1
  9. package/dist/tools/analyze-history.js +89 -5
  10. package/dist/tools/analyze-history.js.map +1 -1
  11. package/dist/tools/audit-accessibility.d.ts +85 -0
  12. package/dist/tools/audit-accessibility.js +4 -1
  13. package/dist/tools/audit-accessibility.js.map +1 -1
  14. package/dist/tools/audit-performance.d.ts +333 -0
  15. package/dist/tools/audit-performance.js +377 -0
  16. package/dist/tools/audit-performance.js.map +1 -0
  17. package/dist/tools/compare-visual-state.d.ts +84 -0
  18. package/dist/tools/generate-e2e-test.d.ts +90 -0
  19. package/dist/tools/generate-e2e-test.js +223 -34
  20. package/dist/tools/generate-e2e-test.js.map +1 -1
  21. package/dist/tools/get-failure.js +2 -1
  22. package/dist/tools/get-failure.js.map +1 -1
  23. package/dist/tools/inspect-page.d.ts +94 -0
  24. package/dist/tools/inspect-page.js +42 -5
  25. package/dist/tools/inspect-page.js.map +1 -1
  26. package/dist/tools/load-report.d.ts +48 -0
  27. package/dist/tools/load-report.js +167 -0
  28. package/dist/tools/load-report.js.map +1 -0
  29. package/dist/tools/probe-script.d.ts +7 -0
  30. package/dist/tools/probe-script.js +266 -7
  31. package/dist/tools/probe-script.js.map +1 -1
  32. package/dist/tools/shared.d.ts +137 -2
  33. package/dist/tools/shared.js +176 -9
  34. package/dist/tools/shared.js.map +1 -1
  35. package/dist/tools/suggest-fix.d.ts +35 -1
  36. package/dist/tools/suggest-fix.js +197 -17
  37. package/dist/tools/suggest-fix.js.map +1 -1
  38. package/dist/tools/validate-selector.d.ts +84 -0
  39. package/dist/tools/validate-selector.js +7 -1
  40. package/dist/tools/validate-selector.js.map +1 -1
  41. package/dist/types/index.d.ts +28 -0
  42. package/dist/utils/project-detector.js +2 -2
  43. package/dist/utils/project-detector.js.map +1 -1
  44. package/dist/utils/report-parser.d.ts +28 -3
  45. package/dist/utils/report-parser.js +77 -11
  46. package/dist/utils/report-parser.js.map +1 -1
  47. package/dist/utils/run-history.d.ts +66 -3
  48. package/dist/utils/run-history.js +149 -3
  49. package/dist/utils/run-history.js.map +1 -1
  50. package/dist/utils/serverless-chromium.d.ts +32 -0
  51. package/dist/utils/serverless-chromium.js +64 -0
  52. package/dist/utils/serverless-chromium.js.map +1 -0
  53. package/package.json +3 -2
  54. package/skill/playwright-e2e-mcp/SKILL.md +71 -15
  55. package/skill/playwright-e2e-mcp/references/setup.md +1 -1
  56. package/skill/playwright-e2e-mcp/references/tools.md +67 -6
package/README.md CHANGED
@@ -9,7 +9,7 @@ Your Playwright test fails. Your AI finds out why, fixes it, and proves the fix
9
9
 
10
10
  <p align="center">
11
11
 
12
- [![CI](https://github.com/xsignon-secure/E2E/actions/workflows/ci.yml/badge.svg)](https://github.com/xsignon-secure/E2E/actions/workflows/ci.yml) [![release](https://img.shields.io/github/v/release/xsignon-secure/E2E)](https://github.com/xsignon-secure/E2E/releases) [![MCP Registry](https://img.shields.io/badge/MCP%20Registry-io.github.xsignon--secure%2FE2E-2563eb)](https://registry.modelcontextprotocol.io/)
12
+ [![npm](https://img.shields.io/npm/v/playwright-e2e-mcp?color=2563eb)](https://www.npmjs.com/package/playwright-e2e-mcp) [![npm downloads](https://img.shields.io/npm/dm/playwright-e2e-mcp?color=2563eb)](https://www.npmjs.com/package/playwright-e2e-mcp) [![CI](https://github.com/xsignon-secure/E2E/actions/workflows/ci.yml/badge.svg)](https://github.com/xsignon-secure/E2E/actions/workflows/ci.yml) [![MCP Registry](https://img.shields.io/badge/MCP%20Registry-io.github.xsignon--secure%2FE2E-2563eb)](https://registry.modelcontextprotocol.io/) [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
13
13
 
14
14
  [![Install in Cursor](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/en/install-mcp?name=playwright-e2e&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInBsYXl3cmlnaHQtZTJlLW1jcCJdfQ==)
15
15
  [![Install in VS Code](https://img.shields.io/badge/VS_Code-Install_server-0098FF?logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect?url=vscode%3Amcp%2Finstall%3F%257B%2522name%2522%253A%2522playwright-e2e%2522%252C%2522command%2522%253A%2522npx%2522%252C%2522args%2522%253A%255B%2522-y%2522%252C%2522playwright-e2e-mcp%2522%255D%257D)
@@ -17,22 +17,49 @@ Your Playwright test fails. Your AI finds out why, fixes it, and proves the fix
17
17
 
18
18
  </p>
19
19
 
20
- ## What it is
20
+ <p align="center">
21
+ <a href="#getting-started">Getting started</a> ·
22
+ <a href="#tools">Tools</a> ·
23
+ <a href="docs/tools.md">Tool reference</a> ·
24
+ <a href="#ci-github-action-and-ci-command">GitHub Action</a> ·
25
+ <a href="#configuration">Configuration</a> ·
26
+ <a href="https://github.com/xsignon-secure/E2E/releases">Changelog</a> ·
27
+ <a href="CONTRIBUTING.md">Contributing</a>
28
+ </p>
21
29
 
22
30
  An [MCP](https://modelcontextprotocol.io) server and agent skill that gives your AI assistant
23
31
  (Claude, Cursor, VS Code Copilot, Gemini, Codex) a full Playwright toolkit: a **test runner**,
24
- a **failure detective** and a **fixer that checks its own work**.
32
+ a **failure detective** and a **fixer that checks its own work**. Other tools let an AI click
33
+ around a browser. This one lets it **keep your test suite green**.
34
+
35
+ ## ❌ Without it / ✅ With it
36
+
37
+ **Without it**, your agent sees `Timeout 5000ms exceeded waiting for getByTestId('checkout')`,
38
+ guesses a new selector, edits the test and hopes. Or it "fixes" the test around a real bug.
25
39
 
26
- Other tools let an AI click around a browser. This one lets it **keep your test suite green**.
40
+ **With it**, your agent reads the page as it was when the test failed, swaps in a locator
41
+ proven unique on that page, re-runs the test, and keeps the edit only if it passes. When the
42
+ button is really gone, it tells you it's an app bug.
27
43
 
28
44
  ```
29
- run-test ──▶ get-failure ──▶ suggest-fix (apply + re-run) ──▶ green
30
- │ │ └─ refuses when the element is really gone (app bug)
31
- │ └─ DOM at failure, failed requests, console errors
32
- └─ background runs, progress ──▶ analyze-history (flaky / broken / regressed)
45
+ run-test ──────┐
46
+ load-report ───┴─▶ get-failure ──▶ suggest-fix (apply + re-run) ──▶ green
47
+ (CI job report) │ └─ refuses when the element is really gone (app bug)
48
+ └─ DOM at failure, failed requests, console errors
49
+ every run ──▶ analyze-history (flaky / broken / slow / which commit broke it)
33
50
  ```
34
51
 
35
- ## What it does
52
+ ## Key features
53
+
54
+ - **Fixes are proven, not guessed.** Every fix is re-run before it's kept, and undone if the test is still red.
55
+ - **Honest.** It won't paper over a real bug to make a test pass.
56
+ - **Works on CI failures too.** Load a CI job's report and traces and get the same diagnosis and fixes.
57
+ - **Flakiness with evidence.** Names the flaky test, whether it's a race, shared state or test order, and the seed that reproduces it.
58
+ - **Lean.** Finding and fixing a broken locator took about **5,300 tokens** versus about **7,400**
59
+ with Playwright MCP plus a shell ([benchmark](#benchmark)).
60
+ - **Free and open source.** MIT, no account, no API key, no telemetry, no paid tier.
61
+
62
+ ## What you can ask
36
63
 
37
64
  | | You ask your AI… | It does this |
38
65
  |---|---|---|
@@ -40,672 +67,256 @@ run-test ──▶ get-failure ──▶ suggest-fix (apply + re-run) ──▶
40
67
  | 🔍 | "Why did this fail?" | Reads Playwright's trace: the page at the moment of failure, the broken request, the console error. No guessing. |
41
68
  | 🛠️ | "Fix it" | Swaps in a locator proven unique on the page, **re-runs the test**, and undoes the edit if it's still red. If the button is really gone, it tells you it's an app bug instead of hiding it. |
42
69
  | 🎲 | "Is this test flaky?" | Re-runs it many ways (repeated, alone, in random order) and names the exact test and seed that cause the flake. |
43
- | 📈 | "What's been breaking lately?" | Keeps a local run history and separates flaky, newly broken and always-broken tests. |
70
+ | 📈 | "What's been breaking lately?" | Keeps a local run history and separates flaky, newly broken and always-broken tests; shows the slowest tests, stability by day and **the commit where each test went bad**. |
71
+ | ☁️ | "Why did CI fail?" | Loads the CI job's report (or merges sharded blob reports), maps its paths onto your checkout and finds the traces in the downloaded artifact, so the same diagnosis and fixes work on CI failures. |
72
+ | ✍️ | "Write a test for this flow" | Walks the steps on the live page first, writes a spec that replays them with assertions on where the flow landed (URL and accessibility snapshot), and runs it once to prove it passes. |
44
73
  | ♿ | "Check this page for accessibility" | Runs axe-core (WCAG 2.2 AA) and returns each problem with a ready-to-paste locator and regression test. |
45
- | 🔐 | "Test the page after login" | Works behind sign-in with saved sessions, headers and click/fill steps. |
74
+ | ⚡ | "Why is this page slow?" | Measures Web Vitals (LCP, CLS, INP, TBT) on desktop or a throttled phone, names the slow element and the elements that jump, and hands back a budget test. |
75
+ | 🧪 | "Show me the error state" | Mocks API responses (500s, empty lists, slow calls) so the AI can inspect, audit or measure states that are hard to reach. |
76
+ | 🔐 | "Test the page after login" | Works behind sign-in with saved sessions, headers and click/fill steps, in Chromium, Firefox or WebKit, and can record or replay network traffic from a HAR file. |
46
77
  | 🤖 | "Do it in CI" | A [GitHub Action and `ci` command](#ci-github-action-and-ci-command) post the diagnosis, flaky verdict and fix diffs on your PR. |
47
78
 
48
- **Why it stands out**
49
-
50
- - **Fixes are proven, not guessed.** Every fix is re-run before it's kept.
51
- - **Honest.** It won't paper over a real bug to make a test pass.
52
- - **Lean.** In the [benchmark](bench/RESULTS.md), finding and fixing a broken locator took about
53
- **5,300 tokens** here versus about **7,400** with Playwright MCP plus a shell.
54
- - **Teaches as it works.** The bundled skill carries a guide to writing tests that stay green
55
- (locators, waiting, auth, test data, mocking, flake patterns, review checklist).
56
- - **Ready in one line.** `npx -y playwright-e2e-mcp init` sets up Claude Code, Cursor and VS Code.
57
-
58
- ### Free and open source, for good
59
-
60
- - **MIT licensed.** Use it, change it, ship it in commercial work; no fees, no paid tier, no feature held back.
61
- - **No account, no API key, no sign-up.** `npx -y playwright-e2e-mcp init` is the whole setup.
62
- - **No telemetry.** It never phones home; your tests, traces and run history stay on your machine.
63
- - **Free dependencies only:** the MCP SDK (Apache-2.0), zod (MIT), axe-core (MPL-2.0) and the Playwright your project already has (Apache-2.0).
64
- - **Free to host yourself**, too: the HTTP bridge deploys to a free Vercel account (see [Hosted endpoint](#hosted-endpoint-chatgpt--remote-clients)).
65
-
66
- <details>
67
- <summary><b>The technical details</b></summary>
68
-
69
- - **Structured runs.** Stats plus each failure's `file:line`, kind (assertion, timeout,
70
- dead dev server, syntax error, crash, full disk…) and a concrete fix hint. Failures that
71
- share one cause are grouped. Long suites run in the background with live progress.
72
- - **Diagnosis from the trace.** The DOM at the moment of failure, the failed action, the
73
- 4xx/5xx requests and console errors, read from Playwright's own trace.
74
- - **Verified fixes.** `suggest-fix` matches a broken locator against the DOM at failure, swaps
75
- in a Playwright locator proven unique on the page
76
- (`getByRole('button', { name: 'Proceed to checkout' })`), re-runs the test and reverts the
77
- edit if it is still red. When nothing on the page plausibly matches, it says the element is
78
- missing and does not patch the test.
79
- - **Flakiness with evidence.** `diagnose-flaky` repeats failing tests in one process and
80
- gives a verdict per test; `isolate: true` re-runs them alone to tell a shared-state bug from
81
- a race inside the test, and `order: true` runs files in random orders to catch an order
82
- dependency, naming the test and the seed that reproduces it. `analyze-history` separates
83
- flaky, regressed and broken tests from the local run record.
84
- - **Accessibility.** `audit-accessibility` runs axe-core against WCAG 2.2 AA on any page,
85
- signed in or after steps, with offending elements as verified locators.
86
- - **Pages behind a login.** Every page tool accepts a Playwright `storageState`, extra
87
- headers, and click/fill steps to reach a modal or a later step.
88
-
89
- </details>
90
-
91
- ### What it does not do
92
-
93
- It doesn't drive an exploratory browsing session step by step, and it doesn't replace
94
- Playwright's own Test Agents inside VS Code. Use Playwright MCP for interactive browsing and
95
- this server for running, diagnosing and fixing the suite. They work side by side.
96
-
97
- ## Demo
98
-
99
- **Hosted endpoint** — what an `initialize` + `tools/list` round-trip against
100
- `https://playwright-e2e-mcp.vercel.app/api/mcp` returns for a client that sends the
101
- bearer token (without one, only `list-tests` and `get-failure` are listed):
102
-
103
- ![Token-protected endpoint: initialize handshake and the tool list](https://raw.githubusercontent.com/xsignon-secure/E2E/main/docs/demo-endpoint.png)
104
-
105
- **A real test run** — `run-test` served over stdio by `npx -y playwright-e2e-mcp`
106
- against the bundled `examples/sample-test.spec.ts` (actual output, unedited):
107
-
108
- ![run-test result: 4 passed, 0 failed, 5.1s](https://raw.githubusercontent.com/xsignon-secure/E2E/main/docs/demo-run-test.png)
109
-
110
- Images are rendered with `node scripts/gen-demo-images.mjs`: the run-test card is real
111
- captured output; the endpoint card is an illustration of the authenticated listing.
112
-
113
- > If this saves you a debugging session, a ⭐ on the repo helps other Playwright users find it.
114
-
115
- ## Install
79
+ ## Requirements
116
80
 
117
- Works with Claude Desktop, Claude Code, Cursor, Windsurf, Codex, Gemini CLI, Freebuff
118
- and every other MCP client — pick whichever route fits:
81
+ - Node.js 20 or newer
82
+ - A project that uses `@playwright/test`, with a browser installed (`npx playwright install chromium`)
83
+ - Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Codex, Gemini CLI or any other MCP client
119
84
 
120
- **Let your agent do it.** Paste this into Claude Code, Cursor, Codex, Gemini CLI or any
121
- other coding agent:
85
+ ## Getting started
122
86
 
123
- > Install the playwright-e2e-mcp server and skill by following
124
- > https://raw.githubusercontent.com/xsignon-secure/E2E/main/llms-install.md
125
-
126
- or run the installer yourself from the project root. It merges the server into each
127
- client's project config it finds (Claude Code, Cursor, VS Code, Gemini CLI, Codex) and
128
- installs the agent skill, leaving your other servers alone:
87
+ **Fastest:** run this in your project root. It adds the server to every client config it finds
88
+ (Claude Code, Cursor, VS Code, Gemini CLI, Codex) and installs the agent skill, leaving your
89
+ other servers alone:
129
90
 
130
91
  ```bash
131
92
  npx -y playwright-e2e-mcp init # --client all | cursor,vscode … · --dry-run
132
93
  ```
133
94
 
134
- | Route | How |
135
- | --- | --- |
136
- | **One command, any project** | `npx -y playwright-e2e-mcp init` |
137
- | **Claude Code plugin (server + skill)** | `/plugin marketplace add xsignon-secure/E2E` then `/plugin install playwright-e2e-mcp@xsignon-secure` |
138
- | **Gemini CLI extension (server + skill)** | `gemini extensions install https://github.com/xsignon-secure/E2E` |
139
- | **Skill only, any agent** | `npx skills add xsignon-secure/E2E` |
140
- | **npm (canonical, fastest)** | `npx -y playwright-e2e-mcp` |
141
- | **MCP Registry** (registry-aware clients discover it automatically) | `io.github.xsignon-secure/E2E` — [listing](https://registry.modelcontextprotocol.io/) |
142
- | **Any client, no npm account needed** | `npx -y github:xsignon-secure/E2E#v0.4.2` (pin a release tag) |
143
- | **Claude Desktop, zero Node setup** | double-click the [`.mcpb` extension](https://github.com/xsignon-secure/E2E/releases) |
144
- | **Remote-only clients (ChatGPT connectors)** | `https://playwright-e2e-mcp.vercel.app/api/mcp` |
145
-
146
- One line for the common clients:
147
-
148
- ```bash
149
- claude mcp add playwright-e2e -- npx -y playwright-e2e-mcp
150
- code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","playwright-e2e-mcp"]}'
151
- ```
152
-
153
- Only need the fix loop? `--tools=core` (or `PW_MCP_TOOLS=core`) exposes six tools and
154
- halves the tool-definition context; see [Configuration](#configuration).
155
-
156
- Details and per-client config: [Installation](#installation) ·
157
- [MCP client configuration](#mcp-client-configuration).
95
+ **Or let your agent do it.** Paste this into any coding agent:
158
96
 
159
- ---
160
-
161
- ## Tools
162
-
163
- | Tool | Purpose |
164
- | --- | --- |
165
- | `run-test` | Run Playwright tests and return stats, failures, diagnostics and hints; reports progress, and `background: true` returns a run id for long suites |
166
- | `get-run-status` | Progress or final result of a background run; can wait for it or cancel it |
167
- | `get-failure` | Deep analysis of one failure: stack, expected/actual, **DOM snapshot at failure (from the Playwright trace)**, failed requests, console errors, next steps |
168
- | `suggest-fix` | Patch a failure: a drifted locator is replaced with one **proven unique** on the DOM at failure, changed copy gets the new expected text; `apply: true` writes it, re-runs the test and reverts if still red |
169
- | `inspect-page` | Open a URL headlessly (optionally signed in, after steps) and return elements with **verified Playwright locators**, CSS selectors, visibility, text and console output |
170
- | `list-tests` | List available tests (`file`, `line`, full title, projects) with filtering |
171
- | `validate-selector` | Check a CSS selector, Playwright selector or **locator expression** against a live page: validity, match count, and a sturdier verified locator for each match |
172
- | `generate-e2e-test` | Scaffold a Playwright test from a description using the project's **real** selectors and verified locators from the live page |
173
- | `compare-visual-state` | Visual regression: screenshot before/after a change and report *what* moved and how colors shifted |
174
- | `audit-accessibility` | Run axe-core (WCAG 2.2 AA by default) on a page, signed in or after steps, and return violations worst first with **verified locators** for the offending elements and a regression check |
175
- | `diagnose-flaky` | Run a failing test 2–10 times **with retries disabled** and return an evidence verdict: `CONSISTENTLY FAILING`, `FLAKY` or `NOT REPRODUCING`; tells parallelism and **test-order** bugs from races |
176
- | `analyze-history` | Rank flaky, regressed and broken tests from the local run history, group failures by cause, or show one test's timeline |
177
-
178
- Two **prompts** package the main workflows for clients that show them as slash commands:
179
- `fix-failing-test` (optional `test`) and `triage-flaky-tests`.
97
+ > Install the playwright-e2e-mcp server and skill by following
98
+ > https://raw.githubusercontent.com/xsignon-secure/E2E/main/llms-install.md
180
99
 
181
- ### Page state: signed-in pages, headers and steps
100
+ **Standard config** works in most clients:
182
101
 
183
- `inspect-page`, `validate-selector` and `compare-visual-state` accept the same page-state
184
- arguments (`suggest-fix` accepts `storageState` and `headers` for a live `url`):
102
+ ```json
103
+ {
104
+ "mcpServers": {
105
+ "playwright-e2e": {
106
+ "command": "npx",
107
+ "args": ["-y", "playwright-e2e-mcp"]
108
+ }
109
+ }
110
+ }
111
+ ```
185
112
 
186
- | Argument | Type | Description |
187
- | --- | --- | --- |
188
- | `storageState` | string | Path inside the project to a Playwright storageState JSON (cookies + localStorage), e.g. `playwright/.auth/user.json` from your setup project. Defaults to `PW_MCP_STORAGE_STATE` |
189
- | `headers` | object | Extra HTTP headers for every request, e.g. `{ "Authorization": "Bearer …" }` |
190
- | `actions` | array | Up to 20 steps run after load: `{ "type": "click" \| "dblclick" \| "hover" \| "fill" \| "press" \| "check" \| "uncheck" \| "select" \| "wait" \| "goto", "locator"?, "value"?, "url"?, "ms"? }`. Locators use Playwright syntax, e.g. `getByRole('button', { name: 'Open settings' })` |
191
- | `viewport` | object | `{ "width": 390, "height": 844 }` (default 1280×720) |
113
+ The server uses its working directory as the project root. If your client starts it somewhere
114
+ else (such as your home directory), add `"env": { "PW_MCP_PROJECT_ROOT": "/path/to/your/project" }`.
192
115
 
193
- A failing step reports which one failed and how many ran. When the page lands on a login
194
- screen, the response says so and suggests `storageState`.
116
+ <details>
117
+ <summary><b>Claude Code</b></summary>
195
118
 
196
- ### `run-test`
119
+ Server and skill, as a plugin:
197
120
 
198
- | Argument | Type | Description |
199
- | --- | --- | --- |
200
- | `projectRoot` | string | Project directory inside the configured root (default: server working directory) |
201
- | `testFiles` | string[] | Files/directories relative to the root; `file:line` supported. Omit to run everything |
202
- | `grep` | string | Only run tests whose title matches this regex |
203
- | `browser` | `chromium` \| `firefox` \| `webkit` | Playwright project to run (matched against config project names) |
204
- | `headed` | boolean | Visible browser window |
205
- | `timeoutMs` | number | Hard wall-clock limit for the run (default `120000`); the whole process tree is killed past it and **partial results are returned** |
206
- | `testTimeoutMs` | number | Per-test timeout passed to Playwright |
207
- | `workers` / `retries` | number | Passed through to Playwright |
208
- | `config` | string | `playwright.config` path **or 1-based index** when the project has several |
209
- | `retryOnFailure` | boolean | Auto-retry failures **once** before reporting them (default `true`; ignored when `retries` is set) |
210
- | `lastFailed` | boolean | Only re-run tests that failed in the previous run (Playwright `--last-failed`) — the fast fix → re-run loop |
211
- | `args` | string[] | Extra Playwright flags from an allowlist (`--repeat-each=N`, `--max-failures=N`, `--update-snapshots`, `--shard=1/3`, `--trace=on`, …); values go after `=` and are checked, and flags that take a path, such as `--config` or `--output`, are rejected |
212
- | `background` | boolean | Return a run id at once; poll `get-run-status` for progress and the result |
213
-
214
- Progress: clients that send a progress token get a notification as each test finishes
215
- (`3/12 tests done, 1 failed`). Every run is appended to the local history used by
216
- `analyze-history`.
217
-
218
- Flakiness handling: by default the server injects `--retries=1` (unless the config
219
- already sets `retries`), so a test that passes on the retry is reported as **flaky**,
220
- not failed. Traces are captured automatically (`--trace=retain-on-failure`) so
221
- `get-failure` can show the DOM at the moment of failure.
222
-
223
- Example result:
224
-
225
- ```markdown
226
- ## Playwright run — ❌ FAILED
227
-
228
- **Command:** `playwright test --config playwright.config.ts tests/checkout.spec.ts --reporter=json`
229
- **duration 4.2s · exit 1 · config `playwright.config.ts`**
230
-
231
- | passed | failed | flaky | skipped | duration |
232
- | ---: | ---: | ---: | ---: | ---: |
233
- | 0 | 1 | 0 | 0 | 1.1s |
234
-
235
- ### ❌ 1 failing test(s)
236
-
237
- ### 1 of 1. checkout.spec.ts › pays with card
238
- **File:** `checkout.spec.ts:5` | **failed · server-unreachable**
239
-
240
- ### ⚠️ SERVER_NOT_RUNNING
241
- Your app (dev server) does not appear to be reachable. Start it in another terminal
242
- (e.g. npm run dev / npm start), keep it running, then retry — or configure `webServer`
243
- in playwright.config.* so Playwright starts it automatically.
121
+ ```
122
+ /plugin marketplace add xsignon-secure/E2E
123
+ /plugin install playwright-e2e-mcp@xsignon-secure
244
124
  ```
245
125
 
246
- ### `get-failure`
247
-
248
- | Argument | Type | Description |
249
- | --- | --- | --- |
250
- | `index` | number | 1-based failure index from the last run (default `1`) |
251
- | `projectRoot` | string | Only used when re-reading the stored report |
252
-
253
- Returns the message/code frame, expected vs actual, stack, failure kind with a
254
- diagnosis, the test's console output, **the DOM snapshot from the Playwright trace
255
- (plus the failed action, its selector, and the action log leading up to it)**,
256
- **the network requests that failed** (4xx/5xx, dead endpoints, no-response — with
257
- method, URL, status and resource type), **the console errors/warnings the page
258
- logged before the failure**, and
259
- numbered next steps (re-run this single test by `file:line`, headed/debug mode,
260
- `validate-selector` when the message mentions a locator, …).
261
-
262
- ### `inspect-page`
126
+ Or the server only:
263
127
 
264
- | Argument | Type | Description |
265
- | --- | --- | --- |
266
- | `url` | string | Full http(s) URL to open (required) |
267
- | `view` | `elements` \| `locators` | `locators` returns a compact map of interactive elements with verified locators (cheapest); `elements` (default) the DOM inventory |
268
- | `projectRoot` | string | Project whose Playwright launches the browser |
269
- | `selector` | string | Inspect matches of this selector instead of the whole DOM |
270
- | `waitFor` | string | Wait for a selector (CSS or `text=…`) before inspecting |
271
- | `waitUntil` | `load` \| `domcontentloaded` \| `networkidle` | Navigation wait condition |
272
- | `includeHtml` | boolean | Include the rendered HTML (capped) |
273
- | `maxHtmlChars` | number | HTML cap, default `20000` |
274
- | `timeoutMs` | number | Overall limit, default `45000` |
275
- | `storageState` / `headers` / `actions` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
276
-
277
- Each element comes with Playwright locators that were **checked on the page** to resolve to
278
- exactly that element, best first (role + accessible name, test id, label, placeholder, text,
279
- then CSS), plus its unique CSS selector, visibility, box and text. Console messages are
280
- captured, errors first. A locator map looks like:
281
-
282
- ```markdown
283
- 1. `getByRole('button', { name: 'Proceed to checkout' })` — button "Proceed to checkout" · also `getByTestId('checkout-btn')`
284
- 2. `getByLabel('Search')` — searchbox "Search"
128
+ ```bash
129
+ claude mcp add playwright-e2e -- npx -y playwright-e2e-mcp
285
130
  ```
286
131
 
287
- ### `list-tests`
288
-
289
- | Argument | Type | Description |
290
- | --- | --- | --- |
291
- | `projectRoot` | string | Project directory |
292
- | `config` | string | Config path or 1-based index |
293
- | `testDir` | string | Restrict scanning to a directory (must stay inside the project) |
294
- | `filter` | string | Case-insensitive substring filter on `file › title` |
295
- | `limit` | number | Max tests returned, default `500` |
132
+ </details>
296
133
 
297
- Uses `playwright test --list` when Playwright works, and **falls back to a source scan**
298
- (keeping the reason) when the install or a spec file is broken.
134
+ <details>
135
+ <summary><b>Claude Desktop</b></summary>
299
136
 
300
- ### `validate-selector`
137
+ Download the `.mcpb` extension from the [latest release](https://github.com/xsignon-secure/E2E/releases/latest)
138
+ and double-click it. It ships its own dependencies, so no Node setup is needed. On install it
139
+ asks for your **project root**: choose the project folder, not your home directory.
301
140
 
302
- | Argument | Type | Description |
303
- | --- | --- | --- |
304
- | `url` | string | Live page to test against (required) |
305
- | `selector` | string | CSS, a Playwright selector (`text=`, `role=`, `>>`) or a locator expression such as `getByRole('button', { name: 'Save' }).first()` (required) |
306
- | `projectRoot` | string | Project whose Playwright launches the browser |
307
- | `timeoutMs` | number | Overall limit, default `45000` |
308
- | `storageState` / `headers` / `actions` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
141
+ You can also add the standard config to `claude_desktop_config.json`.
309
142
 
310
- Locator expressions are parsed into a whitelisted call chain (`getByRole`, `getByText`,
311
- `getByLabel`, `getByPlaceholder`, `getByAltText`, `getByTitle`, `getByTestId`, `locator`,
312
- `first`, `last`, `nth`, `filter`) and are never evaluated as code.
143
+ </details>
313
144
 
314
- Verdicts: `✅ VALID — N matches` with a sample of matches and a **unique locator** for each,
315
- a strict-mode warning when more than one element matches, a hint when a CSS selector is
316
- brittle (`:nth-child`, long descendant chains, XPath), `✅ VALID — 0 matches` with debugging
317
- advice, or `❌ INVALID` with the parse error.
145
+ <details>
146
+ <summary><b>Cursor</b></summary>
318
147
 
319
- ### `generate-e2e-test`
148
+ Click [Install in Cursor](https://cursor.com/en/install-mcp?name=playwright-e2e&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInBsYXl3cmlnaHQtZTJlLW1jcCJdfQ==),
149
+ or go to **Settings → MCP → Add new MCP server** and paste the standard config. For one
150
+ project only, put it in `.cursor/mcp.json`.
320
151
 
321
- | Argument | Type | Description |
322
- | --- | --- | --- |
323
- | `description` | string | What the test should cover (required) |
324
- | `pageUrl` | string | Page the test starts on (default: `baseURL` / `webServer.url` from config) |
325
- | `testDir` / `file` | string | Where to write the spec (default: detected `testDir` + `generated/<slug>.spec.ts`); `file` must end in `.spec.*` or `.test.*` |
326
- | `write` | boolean | Write the file to disk (default `true`) |
327
- | `overwrite` | boolean | Replace an existing spec at the target path; only specs this tool generated can be replaced |
328
- | `liveInspect` | boolean | Cross-check selectors against the live page (default on when a URL is known) |
329
- | `projectRoot` / `config` | string | As with the other tools |
330
-
331
- Reads the agent's recent changes (`git status`, falling back to `git diff HEAD~1`,
332
- then recent mtimes), extracts the locators those files actually declare
333
- (`data-testid`, `getByRole`, `aria-label`, `placeholder`, `id`, `name`, element text),
334
- ranks verified-live selectors first, writes a spec built from them, and reports each
335
- selector with its source `file:line`. When the source has no usable selectors, it uses the
336
- verified locators from the live page that best match the description.
337
-
338
- ### `compare-visual-state`
339
-
340
- | Argument | Type | Description |
341
- | --- | --- | --- |
342
- | `url` | string | Page to capture (required) |
343
- | `name` | string | Baseline id, e.g. `checkout-page` (letters, digits, `. _ -`) |
344
- | `action` | `compare` \| `baseline` | `compare` (default) diffs; `baseline` re-captures the reference |
345
- | `selector` | string | Capture just this element |
346
- | `fullPage` | boolean | Capture the full scrollable page |
347
- | `tolerance` | number | Percent of pixels that may differ (default `0.1`) |
348
- | `pixelThreshold` | number | Per-pixel channel delta considered different (default `60`) |
349
- | `waitUntil` / `waitFor` / `timeoutMs` | — | As with `inspect-page` |
350
- | `storageState` / `headers` / `actions` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
351
-
352
- The first call saves a baseline under `.pw-mcp/visual/` (add that to `.gitignore`, or
353
- commit it for CI comparisons). Later calls report changed-pixel counts, **merged
354
- regions** (`(x, y) 120×40 — 1,200 px`), the **average color shift** ("blue → red"),
355
- and write a red-highlighted diff image for review.
356
-
357
- ### `diagnose-flaky`
358
-
359
- | Argument | Type | Description |
360
- | --- | --- | --- |
361
- | `testFiles` | string[] | Tests to diagnose (`file:line` supported). Defaults to the tests that failed in the most recent run |
362
- | `runs` | number | Repetitions, `2–10` (default `3`) |
363
- | `isolate` | boolean | Also tell parallelism bugs from flakiness: repeat fully parallel (`workers` default `4`), then repeat the failing tests with one worker |
364
- | `order` | boolean | Also run the target files in `runs` random orders with one worker (Playwright 1.64+ `--shuffle`), replay a failing order, and name the test it relies on or is broken by |
365
- | `separateProcesses` | boolean | Start a fresh Playwright process per repetition instead of one `--repeat-each` process (slower; for flakes that depend on global setup or a cold start) |
366
- | `browser` / `headed` / `workers` / `config` | — | As with `run-test` |
367
- | `timeoutMs` | number | Hard wall-clock limit **per repetition** (default `120000`); one `--repeat-each` process gets this × `runs` |
368
- | `projectRoot` | string | Project directory |
369
-
370
- All repetitions run in **one Playwright process** with `--repeat-each` and
371
- `--retries=0`, so each attempt is honest evidence and the browser and workers
372
- start once instead of once per run (about 25% faster on the integration fixture,
373
- more for suites with expensive setup). A small extra reporter records every
374
- attempt, because Playwright's JSON report keeps only one result per test.
375
-
376
- The verdict is **per test**: a table of passes, failures, duration spread and
377
- first error for each test, plus the verdict block for the worst one:
378
-
379
- - **❌ CONSISTENTLY FAILING** — failed every run (same error → reproducible bug,
380
- different errors → still broken, just noisy). Fix it; it is not flaky.
381
- - **⚠️ FLAKY** — some runs passed. Includes `N of M` counts and whether the
382
- failures share one signature (real intermittent bug) or vary (timing/environment
383
- instability).
384
- - **✅ NOT REPRODUCING** — passed every re-run; the original failure was one-off.
385
-
386
- With `isolate: true` a second table compares each failing test **in parallel**
387
- and **alone**:
388
-
389
- | Reading | Meaning |
390
- | --- | --- |
391
- | 🔀 fails only in parallel | Passes every run alone: shared state between tests (same account, DB rows, files, ports). Give each worker its own data or run the file serially |
392
- | ⏱️ flaky even alone | The race is inside the test or the app: missing `await`, asserting before the UI settles |
393
- | ❌ fails alone too | Not a parallelism problem; a real bug |
152
+ </details>
394
153
 
395
- With `order: true` a third section runs the spec files (not just the failing
396
- lines) in random orders with one worker and `--fully-parallel`, so every test is
397
- scheduled on its own. A test that fails in some orders is replayed in a failing
398
- order; if it fails again it is read as **🔗 order dependent**, otherwise as
399
- **flaky regardless of order**:
154
+ <details>
155
+ <summary><b>VS Code / GitHub Copilot</b></summary>
400
156
 
401
- ```
402
- | `tests/cart.spec.ts:10` checkout lists the book | 2/3 | `4` | 🔗 order dependent … |
157
+ Click the **Install in VS Code** button above, or run:
403
158
 
404
- **checkout lists the book**
405
- - Relies on: `adds a book to the cart` (ran before it in every passing order, after it in a failing one)
406
- - Reproduce: `npx playwright test tests/cart.spec.ts --workers=1 --fully-parallel --retries=0 --shuffle=4`
159
+ ```bash
160
+ code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","playwright-e2e-mcp"]}'
407
161
  ```
408
162
 
409
- This is the failure that otherwise reads as "passes alone, fails in the suite".
410
- Set `PW_MCP_SHUFFLE_SEEDS=4,7` on the server to replay given orders, for
411
- example to check a fix.
412
-
413
- Each repetition is recorded in the run history, and the last failing run is
414
- stored, so `analyze-history` and `get-failure` work right afterwards.
415
-
416
- ### `audit-accessibility`
417
-
418
- | Argument | Type | Description |
419
- | --- | --- | --- |
420
- | `url` | string | Page to audit (required) |
421
- | `standard` | `wcag2a` \| `wcag2aa` \| `wcag21aa` \| `wcag22aa` \| `best-practice` \| `all` | Rule set (default `wcag22aa`: WCAG 2.0–2.2 level A and AA; `best-practice` adds axe's extra rules; `all` runs every rule) |
422
- | `minImpact` | `critical` \| `serious` \| `moderate` \| `minor` | Hide violations below this impact |
423
- | `selector` | string | Audit only this part of the page |
424
- | `waitFor` / `waitUntil` / `timeoutMs` | — | As with `inspect-page` (`waitUntil` default `load`, `timeoutMs` default `60000`) |
425
- | `storageState` / `headers` / `actions` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
163
+ </details>
426
164
 
427
- axe-core (the engine behind Lighthouse's accessibility score) is bundled and
428
- evaluated inside the page, so a Content-Security-Policy does not block it.
429
- Example output:
165
+ <details>
166
+ <summary><b>Codex</b></summary>
430
167
 
168
+ ```bash
169
+ codex mcp add playwright-e2e -- npx -y playwright-e2e-mcp
431
170
  ```
432
- ## ♿ 3 accessibility violations (2 critical, 1 serious)
433
171
 
434
- ### 1. critical · `button-name`: Buttons must have discernible text (WCAG 4.1.2)
435
- 1 element · https://dequeuniversity.com/rules/axe/4.14/button-name
436
- - `getByTestId('go')`: Element does not have inner text that is visible to screen readers
172
+ A Playwright run with retries can take longer than Codex's default 60 s tool timeout. Raise
173
+ the timeouts in `~/.codex/config.toml`:
437
174
 
438
- ### 3. serious · `color-contrast`: Elements must meet minimum color contrast ratio thresholds (WCAG 1.4.3)
439
- - `getByText('Low contrast')`: Element has insufficient color contrast of 2.32 (foreground color: #aaaaaa, …). Expected contrast ratio of 4.5:1
175
+ ```toml
176
+ [mcp_servers.playwright-e2e]
177
+ command = "npx"
178
+ args = ["-y", "playwright-e2e-mcp"]
179
+ startup_timeout_sec = 60
180
+ tool_timeout_sec = 600
440
181
  ```
441
182
 
442
- It ends with an `@axe-core/playwright` check for the spec, so the page stays fixed.
183
+ </details>
443
184
 
444
- ### `suggest-fix`
185
+ <details>
186
+ <summary><b>Gemini CLI / Qwen Code</b></summary>
445
187
 
446
- | Argument | Type | Description |
447
- | --- | --- | --- |
448
- | `index` | number | 1-based failure index from the last run (default `1`) |
449
- | `apply` | boolean | Write the fix into the spec (default `false`: return the diff only) |
450
- | `verify` | boolean | After applying, re-run the test and revert the file if it still fails (default `true`) |
451
- | `allowExpectationUpdate` | boolean | Allow applying a changed expected text/value (default `false`, since it can hide a real regression) |
452
- | `url` | string | Heal against this live page instead of the DOM snapshot in the failure trace |
453
- | `storageState` / `headers` | — | For `url`, as in [Page state](#page-state-signed-in-pages-headers-and-steps) |
454
- | `config` / `timeoutMs` / `projectRoot` | — | As with `run-test` |
455
-
456
- Two kinds of fix:
457
-
458
- - **Locator drift.** The broken locator is read from the error, its words are matched
459
- (with synonyms such as *btn → button*, *checkout → proceed to checkout*) against every
460
- element in the DOM at failure, and the best match's verified-unique locator replaces it.
461
- Confidence is `high` when the match is clear, `medium` otherwise.
462
- - **Changed copy.** For `toHaveText`, `toContainText`, `toHaveValue`, `toHaveTitle`,
463
- `toHaveURL` and `toHaveAttribute`, it proposes the value the page actually showed. It is
464
- only applied with `allowExpectationUpdate: true`.
465
-
466
- The response is a unified diff. With `apply: true` the file is written only if it has not
467
- changed since the failure, the single test is re-run (same project, no retries), and the
468
- edit is reverted when it is still red. If nothing on the page shares the locator's
469
- distinctive words, it refuses and reports that the element is most likely **missing**.
470
-
471
- ### `get-run-status`
472
-
473
- | Argument | Type | Description |
474
- | --- | --- | --- |
475
- | `runId` | string | Run id from `run-test` with `background: true` (default: the most recent run) |
476
- | `waitSeconds` | number | Wait up to this long (max 55) for the run to finish |
477
- | `cancel` | boolean | Stop the run and kill its browsers |
188
+ Server and skill, as an extension:
478
189
 
479
- While running it reports tests done / total and failures so far; once finished it returns
480
- the full `run-test` result, which also becomes the last run for `get-failure` and
481
- `suggest-fix`. The 20 most recent runs are kept in memory.
190
+ ```bash
191
+ gemini extensions install https://github.com/xsignon-secure/E2E
192
+ ```
482
193
 
483
- ### `analyze-history`
194
+ Or paste the standard config into `.gemini/settings.json` (Qwen Code: `.qwen/settings.json`).
484
195
 
485
- | Argument | Type | Description |
486
- | --- | --- | --- |
487
- | `view` | `flaky` \| `patterns` \| `test` | `flaky` (default) ranks unstable tests; `patterns` groups failures by kind and normalized error; `test` shows one test's timeline |
488
- | `test` | string | Substring of a test's file or title (filters every view) |
489
- | `lastRuns` | number | How many recent runs to analyze (default `30`) |
490
- | `limit` | number | Max rows (default `15`) |
196
+ </details>
491
197
 
492
- Every `run-test`, `diagnose-flaky` and `suggest-fix` run appends a line to
493
- `<project>/.playwright-e2e-mcp/history.jsonl` (the folder carries its own `.gitignore`;
494
- nothing leaves the machine; `PW_MCP_HISTORY=0` turns it off). The flaky view separates
495
- tests that flip between pass and fail, tests that started failing and stayed red
496
- (regressed), and tests that failed every run (broken), each with a `PFFP~` timeline.
198
+ <details>
199
+ <summary><b>Windsurf, Cline, Zed and other clients</b></summary>
497
200
 
498
- ---
201
+ Add the standard config to the client's MCP config file (Cline: *MCP Servers → Configure*).
202
+ Clients that take a command, such as Zed, run `npx` with the arguments `-y playwright-e2e-mcp`.
499
203
 
500
- ## Installation
204
+ </details>
501
205
 
502
- Requirements:
206
+ <details>
207
+ <summary><b>Freebuff / Codebuff</b></summary>
503
208
 
504
- - Node.js **≥ 20** (the server is built on MCP SDK v2 — the `2026-07-28` spec line)
505
- - A project with `@playwright/test` installed and browsers available
506
- (`npx playwright install chromium`)
209
+ Drop an `mcp.json` with the standard config into your project's `.agents/` folder. This repo
210
+ ships one in [`.agents/mcp.json`](.agents/mcp.json), so opening the checkout attaches the
211
+ server workspace-wide. Freebuff asks you to trust a repository's `.agents/` on first run.
507
212
 
508
- **No npm account needed** — install straight from GitHub (the `prepare` script
509
- builds `dist/` automatically on install). Pin a release tag: an unpinned
510
- `github:xsignon-secure/E2E` runs whatever is on the default branch at that moment.
213
+ </details>
511
214
 
512
- ```bash
513
- npx -y github:xsignon-secure/E2E#v0.4.2
514
- npm install -D github:xsignon-secure/E2E#v0.4.2 @playwright/test # or as a project dependency
515
- npx playwright install chromium
516
- ```
215
+ <details>
216
+ <summary><b>ChatGPT and other remote-only clients</b></summary>
517
217
 
518
- Or grab the packaged tarball from the repo's **GitHub Releases** page and install
519
- it locally:
218
+ Some clients only accept a remote HTTPS server. This repo ships a Streamable HTTP bridge you
219
+ can host for free on Vercel, with OAuth sign-in for claude.ai connectors. See
220
+ [Hosted endpoint](docs/hosted-endpoint.md). On Vercel the page tools (inspect a page, check
221
+ selectors, accessibility and Web Vitals audits) run on a bundled Chromium against public URLs.
222
+ Running and fixing your own tests needs your project, so use the local install for that.
520
223
 
521
- ```bash
522
- npm install -D https://github.com/xsignon-secure/E2E/releases/download/v0.4.2/playwright-e2e-mcp-0.4.2.tgz
523
- ```
524
-
525
- Listed in the **official [MCP Registry](https://registry.modelcontextprotocol.io/)** as
526
- `io.github.xsignon-secure/E2E` — registry-aware clients discover it there, and every
527
- `v*` release tag republishes the entry from CI via [`server.json`](server.json).
224
+ </details>
528
225
 
529
- ### MCP client configuration
226
+ <details>
227
+ <summary><b>Other ways to install</b></summary>
530
228
 
531
- **Claude Code / generic (project-scoped):**
229
+ | Route | How |
230
+ | --- | --- |
231
+ | Skill only, any agent | `npx skills add xsignon-secure/E2E` |
232
+ | MCP Registry | `io.github.xsignon-secure/E2E` ([listing](https://registry.modelcontextprotocol.io/)); registry-aware clients find it there |
233
+ | Straight from GitHub, no npm | `npx -y github:xsignon-secure/E2E#v0.5.1` (pin a release tag) |
234
+ | Release tarball | `npm install -D https://github.com/xsignon-secure/E2E/releases/download/v0.5.1/playwright-e2e-mcp-0.5.1.tgz` |
235
+ | Docker | `docker run -i --rm -v "$PWD:/work" playwright-e2e-mcp` (see [Development](docs/development.md#docker)) |
236
+ | Local checkout | `"command": "node", "args": ["/path/to/E2E/dist/index.js"]` |
532
237
 
533
- ```json
534
- {
535
- "mcpServers": {
536
- "playwright-e2e": {
537
- "command": "npx",
538
- "args": ["-y", "github:xsignon-secure/E2E#v0.4.2"],
539
- "env": { "PW_MCP_PROJECT_ROOT": "/absolute/path/to/your/project" }
540
- }
541
- }
542
- }
543
- ```
238
+ </details>
544
239
 
545
- **Claude Desktop / Cursor / Windsurf / Cline:** add the same block to their MCP config file
546
- (Cline: *MCP Servers → Configure*). **Zed** and other clients that take a command: run `npx`
547
- with the arguments `-y playwright-e2e-mcp`.
548
- The server uses its working directory as the project root; set `PW_MCP_PROJECT_ROOT`
549
- when the client launches it somewhere else (e.g. your home directory).
240
+ ### Your first prompt
550
241
 
551
- **Codex / VS Code / Copilot CLIs:**
242
+ Restart your client, then ask:
552
243
 
553
- ```bash
554
- codex mcp add playwright-e2e -- npx -y github:xsignon-secure/E2E#v0.4.2
555
- code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","github:xsignon-secure/E2E#v0.4.2"]}'
556
244
  ```
557
-
558
- Codex's defaults fight this server: the first launch clones the repo and runs
559
- `tsc` (measured 30 s on a cold `npx` cache, against a 10 s
560
- `startup_timeout_sec` default), and a Playwright run with retries beats the 60 s
561
- `tool_timeout_sec` default. Raise both in `~/.codex/config.toml`:
562
-
563
- ```toml
564
- [mcp_servers.playwright-e2e]
565
- command = "npx"
566
- args = ["-y", "github:xsignon-secure/E2E#v0.4.2"]
567
- startup_timeout_sec = 60
568
- tool_timeout_sec = 600
245
+ Run my Playwright tests and fix whatever fails.
569
246
  ```
570
247
 
571
- **Claude Desktop (one-click):** download and double-click the `.mcpb` Desktop
572
- Extension attached to the [latest release](https://github.com/xsignon-secure/E2E/releases) —
573
- the bundle ships its own dependencies, so no Node setup is required. On install it
574
- asks you to pick your **project root** (required, no default: choose the project
575
- folder, not your home directory) and wires it
576
- into `PW_MCP_PROJECT_ROOT`, so the tools point at a real project from the first call.
248
+ Your agent should run the suite, read the failure, and propose a fix it has re-run. To check the
249
+ setup without your own tests, ask it to run `examples/sample-test.spec.ts` from this repo.
577
250
 
578
- **Claude Code:**
251
+ ## What's new in 0.5.0
579
252
 
580
- ```bash
581
- claude mcp add playwright-e2e -- npx -y github:xsignon-secure/E2E#v0.4.2
582
- ```
583
-
584
- **Gemini CLI / Qwen Code:** paste the `mcpServers` block above into
585
- `.gemini/settings.json` (Qwen Code: `.qwen/settings.json`) — both speak the same
586
- MCP settings format.
587
-
588
- **Freebuff / Codebuff (project-scoped):** this repo ships a committed
589
- [`.agents/mcp.json`](.agents/mcp.json), so opening the checkout in Freebuff
590
- attaches the server workspace-wide — no global config needed. Your own projects
591
- can do the same: drop an `mcp.json` with the block above into their `.agents/`
592
- directory. Freebuff asks you to trust a repository's `.agents/` on first run.
253
+ - **Diagnose CI failures:** `load-report` reads a CI job's report (or sharded blob reports)
254
+ and its traces, so the same diagnosis and verified fixes work on failures from CI.
255
+ - **Know which commit broke a test:** `analyze-history` adds `commits`, `slow` and `trend` views.
256
+ - **Write tests by walking the flow:** `generate-e2e-test` with `steps` runs them on the live
257
+ page, writes the spec with end-state assertions, and runs it once to prove it passes.
258
+ - **Performance audits:** `audit-performance` measures Web Vitals and hands back a budget test.
259
+ - **More page control:** aria snapshots with ready `toMatchAriaSnapshot` assertions, network
260
+ mocks, HAR record/replay, and Chromium, Firefox or WebKit.
593
261
 
594
- All tools ship MCP **tool annotations** (`readOnlyHint`, `destructiveHint`,
595
- `idempotentHint`, `openWorldHint`), so clients can show accurate safety prompts
596
- before running anything.
262
+ [All release notes](https://github.com/xsignon-secure/E2E/releases)
597
263
 
598
- **From a local checkout:**
264
+ ## Tools
599
265
 
600
- ```json
601
- {
602
- "mcpServers": {
603
- "playwright-e2e": {
604
- "command": "node",
605
- "args": ["/path/to/playwright-e2e-mcp/dist/index.js"],
606
- "env": { "PW_MCP_PROJECT_ROOT": "/path/to/your/project" }
607
- }
608
- }
609
- }
610
- ```
266
+ | Tool | Purpose |
267
+ | --- | --- |
268
+ | `run-test` | Run Playwright tests and return stats, failures, diagnostics and hints; reports progress, and `background: true` returns a run id for long suites |
269
+ | `get-run-status` | Progress or final result of a background run; can wait for it or cancel it |
270
+ | `get-failure` | Deep analysis of one failure: stack, expected/actual, **DOM snapshot at failure (from the Playwright trace)**, failed requests, console errors, next steps |
271
+ | `load-report` | Load a run from CI: a JSON report or a folder of sharded blob reports; CI paths are mapped onto the checkout and traces found in the unpacked artifact, so `get-failure`, `suggest-fix` and `diagnose-flaky` work on it, and the run joins the history with its commit |
272
+ | `suggest-fix` | Patch a failure: a drifted locator is replaced with one **proven unique** on the DOM at failure, an ambiguous one (strict mode violation) gets the unique locator of the intended match, changed copy gets the new expected text; `apply: true` writes it, re-runs the test and reverts if still red |
273
+ | `inspect-page` | Open a URL headlessly (optionally signed in, after steps) and return elements with **verified Playwright locators**, CSS selectors, visibility, text and console output, or the **accessibility snapshot with a ready `toMatchAriaSnapshot` assertion** |
274
+ | `list-tests` | List available tests (`file`, `line`, full title, projects) with filtering |
275
+ | `validate-selector` | Check a CSS selector, Playwright selector or **locator expression** against a live page: validity, match count, and a sturdier verified locator for each match |
276
+ | `generate-e2e-test` | Scaffold a Playwright test from a description using the project's **real** selectors and verified locators from the live page, or **walk a flow** step by step on the live page and write a spec that is run once to prove it passes |
277
+ | `compare-visual-state` | Visual regression: screenshot before/after a change and report *what* moved and how colors shifted |
278
+ | `audit-accessibility` | Run axe-core (WCAG 2.2 AA by default) on a page, signed in or after steps, and return violations worst first with **verified locators** for the offending elements and a regression check |
279
+ | `audit-performance` | Measure lab **Web Vitals** (LCP, CLS, TBT, FCP, TTFB, INP during steps), optionally as a throttled phone and the median of several runs; rate them or check budgets, name the LCP and shifting elements as verified locators, list heavy and render-blocking requests, and return a budget test |
280
+ | `diagnose-flaky` | Run a failing test 2–10 times **with retries disabled** and return an evidence verdict: `CONSISTENTLY FAILING`, `FLAKY` or `NOT REPRODUCING`; tells parallelism and **test-order** bugs from races |
281
+ | `analyze-history` | Rank flaky, regressed and broken tests from the local run history, group failures by cause, list the slowest tests, show stability by day, name **the commit where each test went bad**, or show one test's timeline |
611
282
 
612
- ## Hosted endpoint (ChatGPT & remote clients)
283
+ Two **prompts** package the main workflows for clients that show them as slash commands:
284
+ `fix-failing-test` and `triage-flaky-tests`.
613
285
 
614
- Some clients — ChatGPT custom connectors especially — only accept **remote
615
- HTTPS** MCP servers and refuse to spawn a local `npx` process. This repo ships a
616
- Streamable HTTP bridge for exactly that case:
286
+ Every argument and example output is in the **[tool reference](docs/tools.md)**. All tools carry
287
+ MCP tool annotations (`readOnlyHint`, `destructiveHint`, …), so clients can show accurate safety
288
+ prompts.
617
289
 
618
- | | |
619
- | --- | --- |
620
- | **Endpoint** | `https://playwright-e2e-mcp.vercel.app/api/mcp` |
621
- | **Transport** | MCP Streamable HTTP (`POST` JSON in, JSON or SSE out) |
622
- | **Auth** | bearer token (`PW_MCP_HTTP_TOKEN`), or OAuth sign-in with that token as the access key (for claude.ai connectors). The deployment at this URL requires one of them; a bridge started without the variable serves only the read-only tools |
623
- | **Source** | [`api/mcp.ts`](api/mcp.ts) → [`src/http.ts`](src/http.ts) |
624
-
625
- The bridge runs the *same* `createServer()` as the stdio transport; the SDK
626
- serves every request with a fresh server instance, which is what a serverless
627
- function wants. `test/http-bridge.test.mjs` drives the real Node adapter over
628
- `node:http` so a broken bridge fails in CI, not in ChatGPT.
629
-
630
- **This demo deployment is token-protected.** `https://playwright-e2e-mcp.vercel.app`
631
- runs on the maintainer's account and has `PW_MCP_HTTP_TOKEN` set, so a request
632
- without `Authorization: Bearer <token>` (or an OAuth sign-in with it) gets
633
- `401 {"error":"unauthorized"}`. Nothing is held back: the same bridge is free to
634
- host yourself on a free Vercel account, with your own token:
635
-
636
- [![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fxsignon-secure%2FE2E&env=PW_MCP_HTTP_TOKEN&envDescription=A%20random%20secret%20of%2016%2B%20characters%3B%20clients%20send%20it%20as%20a%20bearer%20token%20or%20type%20it%20to%20sign%20in&project-name=playwright-e2e-mcp)
637
-
638
- A serverless function has no browser, so for running and fixing your own tests
639
- the free local install (`npx -y playwright-e2e-mcp init`) is the better route.
640
-
641
- **Open vs. token-protected.** With `PW_MCP_HTTP_TOKEN` set (at least 16
642
- characters; use a random value) every tool is served to clients that send
643
- `Authorization: Bearer <token>`, and every other request gets `401`. Without that
644
- variable anyone who reaches the URL can call the bridge, so it degrades to
645
- `list-tests` and `get-failure` in restricted mode: `list-tests` scans sources
646
- instead of running `playwright test --list` (which would execute the project's
647
- config), callers cannot pick another `projectRoot`, and nothing spawns a process,
648
- drives a browser or writes a file. Child processes started over HTTP get only an
649
- allowlisted environment. `PW_MCP_ALLOWED_HOSTS` (comma separated, `*` for any)
650
- limits the accepted `Host` header; without a token and without that variable,
651
- only `localhost` names and the deployment's own Vercel hostnames are accepted
652
- (DNS-rebinding protection).
653
-
654
- **Add it to ChatGPT:** Settings → Connectors → turn on **Advanced → Developer
655
- mode** → *Create custom connector* → paste the endpoint above. Authentication
656
- **None** only works against a bridge with no token (read-only tools), so for this
657
- deployment choose the connector's **API key** authentication and send
658
- `Authorization: Bearer <token>` — or point the connector at your own deployment
659
- with `PW_MCP_HTTP_TOKEN` unset.
660
-
661
- **Add it to claude.ai (or Claude Desktop's connectors):** Settings → Connectors →
662
- *Add custom connector* → paste the endpoint above and leave the OAuth fields
663
- empty. Claude registers itself and opens a sign-in page; type the deployment's
664
- access key (the value of `PW_MCP_HTTP_TOKEN`) and press **Allow access**. The
665
- bridge is its own small OAuth 2.1 server (discovery, dynamic client
666
- registration, authorization code with PKCE S256, refresh tokens) and keeps no
667
- state: client ids, codes and tokens are signed with a key derived from
668
- `PW_MCP_HTTP_TOKEN`, access tokens last an hour and refresh tokens 30 days, and
669
- changing `PW_MCP_HTTP_TOKEN` signs every connected client out. The sign-in page
670
- shows which site will receive access, so only type the key when that is the
671
- app you are connecting. Source: [`src/oauth.ts`](src/oauth.ts); the
672
- `/.well-known/oauth-*` and `/oauth/*` paths reach the function through the
673
- rewrites in [`vercel.json`](vercel.json).
674
-
675
- To self-host the open, read-only variant, deploy this repo with
676
- `PW_MCP_HTTP_TOKEN` unset; nothing else changes.
677
-
678
- Codex can also take the remote transport instead of spawning `npx`, if you'd
679
- rather not ship Playwright to every machine. Point it at the token through an
680
- environment variable so the secret stays out of the config file:
290
+ **Only need the fix loop?** `--tools=core` (or `PW_MCP_TOOLS=core`) exposes six tools and cuts the
291
+ tool-definition context by about three quarters. Other profiles: `triage`, `inspect`, `author`.
681
292
 
682
- ```bash
683
- export PW_MCP_HTTP_TOKEN=… # ask the operator for the value
684
- codex mcp add playwright-e2e-remote --url https://playwright-e2e-mcp.vercel.app/api/mcp \
685
- --bearer-token-env-var PW_MCP_HTTP_TOKEN
686
- ```
293
+ <details>
294
+ <summary><b>Typical workflow</b></summary>
687
295
 
688
- **What to expect:** `list-tests` works and reports the specs bundled with the
689
- deployment. Even with a token, tools that spawn a browser (`run-test`,
690
- `inspect-page`, `validate-selector`, `diagnose-flaky`, …) cannot download
691
- Chromium in a serverless function, so they return `NO_PLAYWRIGHT` with a note
692
- that points to the local install. Use the stdio install for real runs; the
693
- hosted endpoint is for discovery and for clients that cannot run local
694
- processes.
296
+ 1. `run-test` `{ "testFiles": ["tests/checkout.spec.ts"] }`, or `{ "background": true }`
297
+ for a long suite and `get-run-status` `{ "waitSeconds": 30 }` until it finishes. For a
298
+ failure from CI, download the job's artifacts and call `load-report`
299
+ `{ "report": "ci-artifacts/results.json" }` instead.
300
+ 2. `get-failure` `{ "index": 1 }` to read the code frame, expected/actual, **the DOM
301
+ snapshot at failure**, the failed network requests and the page's console errors.
302
+ 3. `suggest-fix` `{ "index": 1 }` for a locator or copy change: review the diff, then
303
+ `{ "index": 1, "apply": true }` to write it and verify with a re-run. If it reports the
304
+ element is missing, fix the app, not the test.
305
+ 4. For anything it cannot patch: `inspect-page` `{ "url": "…", "view": "locators" }`
306
+ (add `storageState` or `actions` to reach the right screen), then `validate-selector`
307
+ to prove the new locator before editing.
308
+ 5. If a failure looks intermittent: `diagnose-flaky` `{ "runs": 3 }` for fresh evidence,
309
+ and `analyze-history` to see how the test behaved across earlier runs
310
+ (`{ "view": "commits" }` names the commit where it went bad).
311
+ 6. After changing CSS or components: `compare-visual-state` `{ "url": "…", "name": "checkout" }`.
312
+ 7. Re-run **only what failed** with `run-test` `{ "lastFailed": true }` until green.
695
313
 
696
- ```bash
697
- # verify the handshake without any client (the live deployment needs the token;
698
- # drop the authorization header and it answers 401 instead)
699
- curl -X POST https://playwright-e2e-mcp.vercel.app/api/mcp \
700
- -H 'content-type: application/json' \
701
- -H 'accept: application/json, text/event-stream' \
702
- -H "authorization: Bearer $PW_MCP_HTTP_TOKEN" \
703
- -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}}'
704
- ```
314
+ To write a new test, find the locators with `inspect-page` `{ "view": "locators" }`, then call
315
+ `generate-e2e-test` with a `description` and the `steps` of the flow; it walks them on the live
316
+ page, writes the spec and runs it once. To check a page's speed or accessibility, use
317
+ `audit-performance` and `audit-accessibility`.
705
318
 
706
- Redeploy after a change: merge to `main`. Vercel's Git integration deploys
707
- every push, so there is no token to manage and no CLI step — watch the
708
- `Vercel` commit status for the deployment result.
319
+ </details>
709
320
 
710
321
  ## CI: GitHub Action and `ci` command
711
322
 
@@ -733,7 +344,7 @@ steps:
733
344
  - uses: actions/setup-node@v6
734
345
  with: { node-version: 24 }
735
346
  - run: npm ci && npx playwright install --with-deps chromium
736
- - uses: xsignon-secure/E2E@v0.4.2
347
+ - uses: xsignon-secure/E2E@v0.5.1
737
348
  with:
738
349
  flaky-runs: 3 # 0 skips the flaky verdict
739
350
  suggest: 2 # suggest-fix diffs for the first 2 failures
@@ -743,33 +354,25 @@ steps:
743
354
  Inputs: `project-root`, `args`, `flaky-runs`, `suggest`, `comment`,
744
355
  `fail-on-failure`, `version`, `github-token`. Outputs: `exit-code`, `report`.
745
356
 
746
- ## Docker
747
-
748
- ```bash
749
- docker build -t playwright-e2e-mcp .
750
- docker run -i --rm -v "$PWD:/work" playwright-e2e-mcp # stdio MCP server
751
- docker run --rm -v "$PWD:/work" playwright-e2e-mcp ci # CI report
752
- ```
753
-
754
- The image is based on `mcr.microsoft.com/playwright:v1.63.0-noble`, so its
755
- browsers match Playwright 1.63. Mount your project at `/work`; its own
756
- `node_modules` provide `@playwright/test`.
757
-
758
357
  ## Configuration
759
358
 
359
+ <details>
360
+ <summary><b>Environment variables</b></summary>
361
+
760
362
  | Environment variable | Default | Purpose |
761
363
  | --- | --- | --- |
762
364
  | `PW_MCP_PROJECT_ROOT` | server cwd | Default project root for every tool |
763
- | `PW_MCP_TOOLS` | all | Which tools to expose, as tool names and/or profiles, comma separated (same as the `--tools=` flag). Profiles: `core` (run, status, failure, fix, flaky, list), `triage` (run, status, failure, flaky, history, list), `inspect` (page, selector, accessibility, visual), `author` (list, page, selector, generate, run). All twelve tool definitions cost about 5.5k tokens of context per request; `core` costs about 2.2k |
365
+ | `PW_MCP_TOOLS` | all | Which tools to expose, as tool names and/or profiles, comma separated (same as the `--tools=` flag). Profiles: `core` (run, status, failure, fix, flaky, list), `triage` (run, status, failure, CI report, flaky, history, list), `inspect` (page, selector, accessibility, performance, visual), `author` (list, page, selector, generate, run). All fourteen tool definitions cost about 9k tokens of context per request; `core` costs about 2.3k |
764
366
  | `PW_MCP_STORAGE_STATE` | — | Default `storageState` file (inside the project) for the page tools |
765
367
  | `PW_MCP_SHUFFLE_SEEDS` | random | Comma-separated seeds for `diagnose-flaky`'s `order` pass, to replay the orders a diagnosis reported |
766
368
  | `PW_MCP_HISTORY` | on | `0` stops recording runs to `.playwright-e2e-mcp/history.jsonl` |
767
369
  | `PW_MCP_ALLOWED_ROOTS` | — | Extra directories a caller may pass as `projectRoot` (`:`-separated, `;` on Windows). Anything outside these and the default root is rejected |
768
- | `PW_MCP_HTTP_TOKEN` | — | HTTP bridge only: bearer token (16+ characters) that unlocks all tools (see above) |
370
+ | `PW_MCP_HTTP_TOKEN` | — | HTTP bridge only: bearer token (16+ characters) that unlocks all tools (see [Hosted endpoint](docs/hosted-endpoint.md)) |
769
371
  | `PW_MCP_ALLOWED_HOSTS` | localhost + Vercel hostnames when there is no token | HTTP bridge only: comma-separated `Host` allowlist; `*` accepts any |
770
372
  | `PW_MCP_PASSTHROUGH_ENV` | — | HTTP bridge only: comma-separated extra variables passed to test runs (e.g. `BASE_URL`) |
771
373
  | `PW_MCP_MAX_CHILDREN` | `4` | HTTP bridge only: how many test runs and browser probes may run at once |
772
374
  | `PW_MCP_BLOCK_PRIVATE_URLS` | off (on for the HTTP bridge) | `1` makes the URL tools refuse loopback, private-network and cloud-metadata addresses, including redirects and subresources |
375
+ | `PW_MCP_SERVERLESS_CHROMIUM` | on for Vercel | Launch the page tools on the bundled serverless Chromium (`@sparticuz/chromium`, Linux x64, Node 22+); `0` turns it off, `1` turns it on elsewhere |
773
376
  | `LOG_LEVEL` / `MCP_LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` \| `silent`; `LOG_LEVEL` wins when both are set |
774
377
  | `LOG_FORMAT` / `MCP_LOG_FORMAT` | `text` | `text`, or `json` (also `ndjson`) for one structured JSON object per line — ready for a log ingester |
775
378
 
@@ -782,24 +385,7 @@ object (`time`, `level`, `message` plus context):
782
385
  {"time":"2026-10-07T08:00:00.000Z","level":"info","message":"server ready","name":"playwright-e2e-mcp"}
783
386
  ```
784
387
 
785
- ## Typical workflow
786
-
787
- 1. `run-test` `{ "testFiles": ["tests/checkout.spec.ts"] }`, or `{ "background": true }`
788
- for a long suite and `get-run-status` `{ "waitSeconds": 30 }` until it finishes.
789
- 2. `get-failure` `{ "index": 1 }` to read the code frame, expected/actual, **the DOM
790
- snapshot at failure**, the failed network requests and the page's console errors.
791
- 3. `suggest-fix` `{ "index": 1 }` for a locator or copy change: review the diff, then
792
- `{ "index": 1, "apply": true }` to write it and verify with a re-run. If it reports the
793
- element is missing, fix the app, not the test.
794
- 4. For anything it cannot patch: `inspect-page` `{ "url": "…", "view": "locators" }`
795
- (add `storageState` or `actions` to reach the right screen), then `validate-selector`
796
- to prove the new locator before editing.
797
- 5. If a failure looks intermittent: `diagnose-flaky` `{ "runs": 3 }` for fresh evidence,
798
- and `analyze-history` to see how the test behaved across earlier runs.
799
- 6. After changing CSS or components: `compare-visual-state` `{ "url": "…", "name": "checkout" }`.
800
- 7. Re-run **only what failed** with `run-test` `{ "lastFailed": true }` until green.
801
-
802
- To write a new test, start with `generate-e2e-test` `{ "description": "checkout with a saved card" }`.
388
+ </details>
803
389
 
804
390
  ## Benchmark
805
391
 
@@ -819,306 +405,34 @@ apply) and with Playwright MCP plus a shell (`playwright test`, `browser_navigat
819
405
  Tokens are characters ÷ 4. The Playwright MCP route is a lower bound: it assumes the agent
820
406
  picks the right locator from the snapshot on the first try, which the server does for it here.
821
407
 
822
- ## Edge cases handled
823
-
824
- | Situation | Behaviour |
825
- | --- | --- |
826
- | No Playwright installed | `NO_PLAYWRIGHT` error with the exact install commands for your package manager |
827
- | Dev server not running | Failure classified `server-unreachable` / `SERVER_NOT_RUNNING` with a "start your dev server" hint (and `webServer` advice) |
828
- | Test exceeds `timeoutMs` | Process **group** is killed (SIGINT→SIGKILL on POSIX, `taskkill /T /F` on Windows) and partial results are returned |
829
- | Browser crashes | Classified `browser-crash` with retry / reinstall guidance |
830
- | Windows backslashes | All paths normalized lexically (`C:\a\..\b` → `C:/b`); unit-tested on both platforms |
831
- | Flaky tests | Failing tests are automatically retried once (`--retries=1`) before being reported; passes surface as **flaky** with a stability warning; `diagnose-flaky` decides flaky-vs-broken with multi-run evidence |
832
- | Trace/DOM context | `--trace=retain-on-failure` is passed automatically, so `get-failure` can show the exact DOM at the moment of failure — plus the failed network requests (`*.network` logs) and console errors from the same trace |
833
- | Slow re-runs after a fix | `run-test` with `lastFailed: true` re-runs only the tests that failed last time (`--last-failed`) |
834
- | Several `playwright.config` files | Returns a numbered menu (`MULTIPLE_CONFIGS`); pick with `config: "2"` or a path |
835
- | Syntax error in a spec | `SYNTAX_ERROR` with file:line; nothing crashes; `list-tests` falls back to a source scan |
836
- | MCP client disconnects | Per-request `AbortSignal` kills the run; stdin end triggers shutdown, and every tracked child tree is force-killed (`killActiveChildren`) |
837
- | Disk full | `ENOSPC` detected → `DISK_FULL` with a "free space" hint; logging never throws |
838
- | Malicious paths | `../../etc/passwd`, absolute paths outside the root, a `projectRoot` outside the allowed roots, symlinks that leave the root, URLs and null bytes are rejected with `INVALID_PATH`; extra CLI args must be allowlisted Playwright flags |
839
-
840
- ## Security notes
841
-
842
- - **No shell**: Playwright is spawned as `node <playwright/cli.js> …` with an argument
843
- array — no command interpolation. Extra `args` are limited to an allowlist of
844
- Playwright flags, each with a checked value (`--flag=value`), so a caller cannot
845
- swap in another `--config` or `--output` or hand git an option through
846
- `--only-changed`. Paths with a segment starting with `-` are rejected, so a test
847
- file name cannot be read as a flag.
848
- - **Path sandbox**: user paths must stay inside the project root, checked lexically
849
- and again with symlinks resolved. A caller-supplied `projectRoot` must sit inside
850
- `PW_MCP_PROJECT_ROOT` (or a `PW_MCP_ALLOWED_ROOTS` entry).
851
- - **Generated code**: `generate-e2e-test` only writes `*.spec.*` / `*.test.*` files,
852
- only overwrites specs whose header it wrote itself, never writes through a symlink,
853
- and escapes every value it puts into strings or comments.
854
- - **HTTP bridge**: read-only tools unless `PW_MCP_HTTP_TOKEN` is set; child processes
855
- started over HTTP get only an allowlisted environment (`PATH`, `HOME`, temp dirs,
856
- locale, `CI`, `PLAYWRIGHT_*`, `npm_config_*` without embedded credentials, plus
857
- anything in `PW_MCP_PASSTHROUGH_ENV`), at most `PW_MCP_MAX_CHILDREN` run at once
858
- (a disconnected client frees its slot immediately), and internal error messages
859
- are not returned to clients. The allowlist only covers the child's own
860
- environment: test code runs as the same OS user, so on Linux it could still read
861
- the server's startup environment from `/proc`. That is why only token holders
862
- can run project code; keep other secrets out of the bridge's environment, or run
863
- it under a separate user.
864
- - **SSRF guard**: on the HTTP bridge (or with `PW_MCP_BLOCK_PRIVATE_URLS=1`) the URL tools
865
- refuse hosts that resolve to loopback, private, link-local/metadata or reserved
866
- addresses, and route the browser through a local proxy that applies the same check
867
- to every redirect hop and subresource; IPv6 forms that embed an IPv4 address
868
- (mapped, NAT64, 6to4, Teredo) are blocked too, and WebRTC UDP is disabled so a
869
- page cannot reach the network around the proxy. It is off for stdio by default because
870
- opening `http://localhost` dev servers is what these tools are for.
871
- - **Cleanup**: temp report/script files are written to the OS temp dir and removed;
872
- child processes are tracked and killed on shutdown. Trace decompression has one
873
- size budget per archive, and PNGs are capped at 16,384 px per side and 50 M pixels.
874
-
875
- ## Development
876
-
877
- ```
878
- src/
879
- ├── index.ts # bin entry point (--version/--help, main-module guard)
880
- ├── server.ts # McpServer setup, tool registration, shutdown handling
881
- ├── tools/ # the twelve tools, the probe script + shared plumbing
882
- ├── prompts.ts # fix-failing-test, triage-flaky-tests
883
- ├── utils/ # playwright-runner, report-parser, project-detector, path-utils,
884
- │ # logger, trace-reader (trace.zip → DOM/network/console),
885
- │ # image-diff (PNG codec + pixel diff), change-analyzer,
886
- │ # locator-expr, run-history, run-registry
887
- └── types/ # shared interfaces and the ErrorKind taxonomy
888
- ```
889
-
890
- Built on **`@modelcontextprotocol/server` v2** (the `2026-07-28` MCP spec line) with
891
- Zod v4 standard schemas; every tool declares spec tool annotations.
892
-
893
- ```bash
894
- npm install
895
- npm run build # tsc → dist/ (zero errors)
896
- npm test # build + test/run-tests.mjs (unit tests, any Node ≥20)
897
- npm run e2e # build + e2e/run.mjs: live MCP ↔ Playwright integration suite
898
- npm run mcpb # build + bundle dist/ into playwright-e2e-mcp-<version>.mcpb (+ .sha256)
899
- npm run mcpb:smoke # ...then install that bundle in a temp dir, launch it and assert the handshake
900
- npm run mcpb:sign # ...then sign it with MCPB_CERT/MCPB_KEY → <version>.signed.mcpb
901
- npm run skill:zip # just the archive: dist-skill/playwright-e2e-mcp.zip
902
- npm run skill:package # the uploadable folder: ../agensi-skill-<version>/ (skill copy, README, zip)
903
- npm run skill:check # build, then fail if a packaged copy has drifted from skill/
904
- ```
905
-
906
- Tests cover the report parser (sample Playwright JSON, trace attachments), path utils
907
- (Windows and macOS paths, sandboxing), the project detector (temp-dir fixtures: config
908
- discovery, multiple configs, missing install, test-file scanning), the shared tool
909
- helpers, the **trace reader** (synthetic trace.zip: error, failed action, DOM snapshot,
910
- `*.network` failed-request parsing, console error/warning events), the **image diff**
911
- (PNG round-trip, regions, color shift, dimension changes), the **change analyzer**
912
- (selector extraction, git + mtime paths), the **flaky verdict logic**
913
- (failure signatures, CONSISTENTLY FAILING / FLAKY / NOT REPRODUCING / NO TESTS RAN,
914
- per-test grouping of repeat attempts and the parallel-vs-alone reading), the `ci`
915
- argument parser and tool profiles,
916
- the **HTTP bridge** (token, restricted mode, Host allowlist), the **security
917
- regressions** (sandbox escapes, argument smuggling, symlink writes, code injection,
918
- SSRF ranges, env scrubbing, decode limits), the `.mcpb` manifest, the `.mcpb` zip
919
- extractor, the **skill package** (byte-identical rebuilds, a CRLF checkout hashing the
920
- same, refusal of junk/binary/symlinks, and drift detection between `skill/` and a
921
- packaged copy), and the MCP config files.
922
-
923
- ### Integration suite (`npm run e2e`)
924
-
925
- Unit tests prove the logic; the integration suite proves the loop. It boots the real
926
- server over stdio against a live fixture app and a Playwright project under
927
- `e2e/fixture/`, then drives it exactly like an MCP client and asserts ~40 behaviours
928
- that only appear end-to-end:
929
-
930
- - initialize handshake, 12 tools, 2 prompts, spec tool annotations and object input schemas,
931
- - verified locators, locator validation, signed-in pages (`storageState`) and page steps,
932
- - background runs with progress, run history views, and `suggest-fix` healing a drifted
933
- locator and verifying it with a re-run (and refusing when the element is missing),
934
- - live DOM inspection, CSS selector validation (matches, zero matches, engine
935
- syntax, parse errors), dead-server detection,
936
- - visual regression: baseline → unchanged compare → `blue → red` diff detection,
937
- - `run-test` pass/fail/`lastFailed` stats and meta lines,
938
- - `get-failure` trace diagnostics: DOM at failure, the 404 network request,
939
- the `console.error` message, diagnosis and next steps,
940
- - auto-retry turning a first-run failure into `PASSED (1 flaky)`,
941
- - `diagnose-flaky` verdict **FLAKY** (2 of 3 runs) with retries disabled, in one
942
- `--repeat-each` process, and `isolate` reading a shared-slot test as "fails only in parallel",
943
- - `--tools=` profiles trimming `tools/list`, and the `ci` command's exit code, report,
944
- `--output` file and GitHub job summary,
945
- - `generate-e2e-test` writing its scaffold, plus error paths
946
- (missing test path, unknown tool, unreachable server).
947
-
948
- First run needs the browser once: `npx playwright install chromium`.
949
- CI runs the suite on Ubuntu and Windows (see `.github/workflows/ci.yml`).
950
-
951
- ### Signing the bundle
952
-
953
- ```bash
954
- MCPB_CERT=~/signing/cert.pem MCPB_KEY=~/signing/key.pem npm run mcpb:sign
955
- ```
956
-
957
- Signing appends a PKCS#7 signature block to the bundle, so the signed file is
958
- *the reproducible bundle plus that block*. That ordering is the point: the bytes
959
- CI proved identical across platforms stay inside the signed file, and
960
- `mcpb unsign` returns them exactly. `scripts/sign-mcpb.mjs` asserts that round
961
- trip on every run and deletes its output if it fails, which is what keeps
962
- `<version>.signed.mcpb` tied to `<version>.mcpb`.
963
-
964
- The private key is read from a path you pass in (`--cert`/`--key` or
965
- `MCPB_CERT`/`MCPB_KEY`) and never from the repository — keep it outside version
966
- control, and back it up: the same key across releases is what makes the signer
967
- recognisable.
968
-
969
- What signing does and does not buy you is decided by the certificate, not by
970
- this script. `mcpb verify` and `mcpb info` accept a signature only when the
971
- certificate chains to something in the OS trust store, so with a self-signed
972
- certificate a host still reports **Not signed** — the signature is real
973
- integrity and key continuity, verifiable by the fingerprint the script prints,
974
- but not third-party identity. A certificate issued by a public CA, or an
975
- internal one installed on your machines, is what makes hosts report it as
976
- signed.
977
-
978
- ### Assembling a release
979
-
980
- ```bash
981
- npm run mcpb && npm pack # the bundle, then the npm tarball
982
- npm run release:assemble -- --tag v0.2.3 # sign, checksum and write the notes
983
- npm run release:publish -- --tag v0.2.3 --assets dist-release/assets \
984
- --title-file dist-release/RELEASE_TITLE.txt \
985
- --body-file dist-release/RELEASE_BODY.md --download dist-release/downloaded
986
- npm run release:verify -- v0.2.3 --dir dist-release/downloaded
987
- npm run release:publish -- --tag v0.2.3 --publish
988
- ```
989
-
990
- Pushing a `v*` tag runs exactly that sequence in
991
- `.github/workflows/release.yml`. The release used to be assembled by hand, and
992
- the notes were written from what those commands printed: every hash in them was
993
- a copy, and nothing compared the bundle hash printed there with the bundle that
994
- shipped.
995
-
996
- `scripts/assemble-release.mjs` stages the whole release into
997
- `dist-release/assets/`: the bundle and its sidecar, the **signed** bundle and the
998
- signing certificate when a key is configured, the npm tarball, the skill package
999
- built by the same writer as `npm run skill:zip`, and `SHA256SUMS.txt` covering
1000
- every file it staged. It empties that directory first, so a file left over from
1001
- an earlier run cannot be hashed and published as if this run had produced it.
1002
-
1003
- The notes are half written, half generated. `release-notes/<tag>.md` holds the
1004
- prose — its H1 becomes the release title and has to name the tag — and the script
1005
- appends the asset table, the reproduction recipe carrying the real bundle hash,
1006
- the signer and fingerprint, and the commit list. The prose is held to the same
1007
- rule: a 64-character hex string in it has to be a hash this release publishes,
1008
- and writing an `### Assets` section of your own fails the assembly rather than
1009
- shipping a second, staler table. See [release-notes/README.md](release-notes/README.md).
1010
-
1011
- `scripts/publish-release.mjs` fills the release **as a draft**, uploads the
1012
- assets, downloads every one of them back and compares it byte for byte, and only
1013
- `--publish` makes it public — after `release:verify` has re-derived the release
1014
- from the tag. Re-running is safe: a draft this workflow assembled is refreshed in
1015
- place, while a release it did not assemble, or one already published, is refused
1016
- by name instead of overwritten.
1017
-
1018
- Signing is optional and its absence is stated rather than hidden: without the
1019
- `MCPB_KEY` and `MCPB_CERT` repository secrets the release is assembled and
1020
- published **unsigned**, with a notice in the run and a line saying so in the
1021
- notes. Add both — the private key stays out of the repository and out of the logs:
1022
-
1023
- ```bash
1024
- gh secret set MCPB_KEY --repo xsignon-secure/E2E < ~/signing/key.pem
1025
- gh secret set MCPB_CERT --repo xsignon-secure/E2E < ~/signing/cert.pem
1026
- ```
1027
-
1028
- Creating the release and uploading its assets needs `contents: write`, which the
1029
- job requests for itself; if it is refused, the run names the setting to change.
408
+ ## Demo
1030
409
 
1031
- ### Verifying a published release
410
+ A real `run-test` call served over stdio by `npx -y playwright-e2e-mcp` against the bundled
411
+ `examples/sample-test.spec.ts` (actual output, unedited):
1032
412
 
1033
- ```bash
1034
- git checkout v0.4.2 && npm ci && npm run mcpb
1035
- npm run release:verify -- v0.4.2
1036
- ```
413
+ ![run-test result: 4 passed, 0 failed, 5.1s](https://raw.githubusercontent.com/xsignon-secure/E2E/main/docs/demo-run-test.png)
1037
414
 
1038
- The release notes promise that the tag rebuilds the published bundle byte for
1039
- byte. `scripts/verify-release.mjs` checks that promise from a checkout of the
1040
- tag: the published bundle against your fresh build, every checksum the release
1041
- publishes (and that nothing shippable is missing from `SHA256SUMS.txt`), a
1042
- **signed** bundle against the bytes its signature wraps, the **skill package**
1043
- against a rebuild from the same tag, and that the published archive still
1044
- extracts with that version's manifest. The skill package is required of a tag
1045
- whose tree carries the publishing workflow; older tags predate it, so there it
1046
- is checked when present and skipped when not, like the signing certificate.
1047
- `.github/workflows/verify-release.yml` runs it on demand for any past tag, and by
1048
- itself for a release published by hand. A release assembled by
1049
- `.github/workflows/release.yml` runs the same check on its own assets *before*
1050
- it is published — GitHub does not raise `release: published` for a release created
1051
- with the repository's own token, so the check is a step there rather than a
1052
- trigger. The `.tgz` is checked for download integrity only: npm
1053
- records the packer's line endings, so a tarball packed locally on another OS is
1054
- legitimately different.
1055
-
1056
- ### Packaging the agent skill
415
+ ## Free and open source
1057
416
 
1058
- ```bash
1059
- npm run skill:package # ../agensi-skill-<version>/, ready to upload
1060
- npm run skill:zip # only the archive: dist-skill/playwright-e2e-mcp.zip
1061
- npm run skill:check # build, then compare every copy
1062
- npm run skill:check -- ../agensi-skill-0.3.0
1063
- ```
417
+ - **MIT licensed.** Use it, change it, ship it in commercial work; no fees, no paid tier, no feature held back.
418
+ - **No account, no API key, no sign-up.** `npx -y playwright-e2e-mcp init` is the whole setup.
419
+ - **No telemetry.** It never phones home; your tests, traces and run history stay on your machine.
420
+ - **Free dependencies only:** the MCP SDK (Apache-2.0), zod (MIT), axe-core (MPL-2.0) and the Playwright your project already has (Apache-2.0).
421
+ - **Free to host yourself**, too: the HTTP bridge deploys to a free Vercel account (see [Hosted endpoint](docs/hosted-endpoint.md)).
1064
422
 
1065
- `skill/` is an [Agent Skill](https://agentskills.io/specification) — the
1066
- instructions an agent follows to drive this server — and the marketplace gets
1067
- the same Markdown as a zip with `playwright-e2e-mcp/` at its root.
1068
- `scripts/package-skill.mjs` produces that zip, so the uploaded copy is built
1069
- rather than assembled by hand, and it refuses what the marketplace rejects: a
1070
- missing `SKILL.md`, a non-text file, a symlink, `Thumbs.db`.
1071
-
1072
- `skill:package` writes the whole folder a release would otherwise assemble by
1073
- hand — the skill copy, `skill/README.md` and the archive — to
1074
- `../agensi-skill-<version>` (or `--out <dir>`), then verifies it with the same
1075
- comparison `check` uses, so a successful exit means the folder really is the
1076
- skill tree. It is idempotent: a folder that already matches is left untouched.
1077
- A folder that has drifted is refused instead of overwritten, because it may
1078
- already be published — bump `version` for a new package, or pass `--force` to
1079
- replace that one.
1080
-
1081
- The archive comes from the same deterministic writer as the `.mcpb`, so the
1082
- same tree always hashes the same, and text payloads are LF-normalised first:
1083
- a CRLF checkout builds the bytes an LF one does. CI runs `skill:check`, generates
1084
- a package folder, and uploads the built zip as an artifact
1085
- (`.github/workflows/ci.yml`).
1086
-
1087
- **Every release publishes the package too.** Assembling a release builds it from
1088
- the release tag and attaches it as `<name>-<version>-skill.zip`, and the
1089
- `.github/workflows/release.yml` run downloads that asset back off the draft and
1090
- byte-compares it with the build before anything is published. The marketplace
1091
- upload is that asset, not a folder someone assembled — a deterministic archive
1092
- means "the same build as the tagged release" is an exact statement, and
1093
- `release:verify` checks it again against a rebuild from the tag.
1094
- `.github/workflows/publish-skill-package.yml` stays for the repair path: run it
1095
- by hand to attach or rebuild the package for a release that is missing it.
1096
-
1097
- `skill:check` is the other half of that, and the part that matters when a
1098
- package has already been uploaded. It builds twice (the bytes must agree),
1099
- reads the archive back and compares its contents with `skill/` file by file,
1100
- then compares a packaged copy — `../agensi-skill-<version>` by default, or a
1101
- path you pass: the package folder, the bare `playwright-e2e-mcp/` folder, or
1102
- the zip on its own. Comparisons are by content, never by archive bytes, because
1103
- a zip written by a platform `zip` tool carries real timestamps that say nothing
1104
- about whether the Markdown inside it is current. Any file that differs, is
1105
- missing or is extra is named on stderr and the command exits 1 — the repository
1106
- copy and the uploaded copy are the same document, so an edit to either one
1107
- should not be invisible.
1108
-
1109
- ### Try the example
1110
-
1111
- With Playwright installed in your project:
423
+ **What it does not do:** it doesn't drive an exploratory browsing session step by step, and it
424
+ doesn't replace Playwright's own Test Agents inside VS Code. Use Playwright MCP for interactive
425
+ browsing and this server for running, diagnosing and fixing the suite. They work side by side.
1112
426
 
1113
- ```bash
1114
- npx playwright test examples/sample-test.spec.ts
1115
- ```
427
+ ## More documentation
1116
428
 
1117
- or ask your agent to call `run-test` with
1118
- `"testFiles": ["examples/sample-test.spec.ts"]` — it hits the public
1119
- [example.com](https://example.com) page, so it verifies browsers, network and the MCP
1120
- pipeline in one shot.
429
+ - [Tool reference](docs/tools.md): every argument, example outputs, edge cases handled
430
+ - [Hosted endpoint](docs/hosted-endpoint.md): the HTTP bridge for ChatGPT and claude.ai connectors
431
+ - [Security](SECURITY.md): the security model and how to report a vulnerability
432
+ - [Development](docs/development.md): building, tests, Docker, signing and releasing
433
+ - [Contributing](CONTRIBUTING.md) · [Code of conduct](CODE_OF_CONDUCT.md)
1121
434
 
1122
435
  ## License
1123
436
 
1124
- MIT
437
+ [MIT](LICENSE). If this saves you a debugging session, a ⭐ on the repo helps other Playwright
438
+ users find it.