playwright-e2e-mcp 0.5.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +227 -1055
- package/dist/http.d.ts +1 -0
- package/dist/http.js +1 -0
- package/dist/http.js.map +1 -1
- package/dist/tools/probe-script.js +10 -0
- package/dist/tools/probe-script.js.map +1 -1
- package/dist/tools/shared.js +25 -2
- package/dist/tools/shared.js.map +1 -1
- package/dist/utils/project-detector.js +2 -2
- package/dist/utils/project-detector.js.map +1 -1
- package/dist/utils/serverless-chromium.d.ts +32 -0
- package/dist/utils/serverless-chromium.js +64 -0
- package/dist/utils/serverless-chromium.js.map +1 -0
- package/package.json +3 -2
- package/skill/playwright-e2e-mcp/SKILL.md +2 -2
- package/skill/playwright-e2e-mcp/references/setup.md +1 -1
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@ Your Playwright test fails. Your AI finds out why, fixes it, and proves the fix
|
|
|
9
9
|
|
|
10
10
|
<p align="center">
|
|
11
11
|
|
|
12
|
-
[](https://www.npmjs.com/package/playwright-e2e-mcp) [](https://www.npmjs.com/package/playwright-e2e-mcp) [](https://github.com/xsignon-secure/E2E/actions/workflows/ci.yml) [](https://registry.modelcontextprotocol.io/) [](LICENSE)
|
|
13
13
|
|
|
14
14
|
[](https://cursor.com/en/install-mcp?name=playwright-e2e&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInBsYXl3cmlnaHQtZTJlLW1jcCJdfQ==)
|
|
15
15
|
[](https://insiders.vscode.dev/redirect?url=vscode%3Amcp%2Finstall%3F%257B%2522name%2522%253A%2522playwright-e2e%2522%252C%2522command%2522%253A%2522npx%2522%252C%2522args%2522%253A%255B%2522-y%2522%252C%2522playwright-e2e-mcp%2522%255D%257D)
|
|
@@ -17,22 +17,49 @@ Your Playwright test fails. Your AI finds out why, fixes it, and proves the fix
|
|
|
17
17
|
|
|
18
18
|
</p>
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
<p align="center">
|
|
21
|
+
<a href="#getting-started">Getting started</a> ·
|
|
22
|
+
<a href="#tools">Tools</a> ·
|
|
23
|
+
<a href="docs/tools.md">Tool reference</a> ·
|
|
24
|
+
<a href="#ci-github-action-and-ci-command">GitHub Action</a> ·
|
|
25
|
+
<a href="#configuration">Configuration</a> ·
|
|
26
|
+
<a href="https://github.com/xsignon-secure/E2E/releases">Changelog</a> ·
|
|
27
|
+
<a href="CONTRIBUTING.md">Contributing</a>
|
|
28
|
+
</p>
|
|
21
29
|
|
|
22
30
|
An [MCP](https://modelcontextprotocol.io) server and agent skill that gives your AI assistant
|
|
23
31
|
(Claude, Cursor, VS Code Copilot, Gemini, Codex) a full Playwright toolkit: a **test runner**,
|
|
24
|
-
a **failure detective** and a **fixer that checks its own work**.
|
|
32
|
+
a **failure detective** and a **fixer that checks its own work**. Other tools let an AI click
|
|
33
|
+
around a browser. This one lets it **keep your test suite green**.
|
|
34
|
+
|
|
35
|
+
## ❌ Without it / ✅ With it
|
|
25
36
|
|
|
26
|
-
|
|
37
|
+
**Without it**, your agent sees `Timeout 5000ms exceeded waiting for getByTestId('checkout')`,
|
|
38
|
+
guesses a new selector, edits the test and hopes. Or it "fixes" the test around a real bug.
|
|
39
|
+
|
|
40
|
+
**With it**, your agent reads the page as it was when the test failed, swaps in a locator
|
|
41
|
+
proven unique on that page, re-runs the test, and keeps the edit only if it passes. When the
|
|
42
|
+
button is really gone, it tells you it's an app bug.
|
|
27
43
|
|
|
28
44
|
```
|
|
29
|
-
run-test
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
45
|
+
run-test ──────┐
|
|
46
|
+
load-report ───┴─▶ get-failure ──▶ suggest-fix (apply + re-run) ──▶ green
|
|
47
|
+
(CI job report) │ └─ refuses when the element is really gone (app bug)
|
|
48
|
+
└─ DOM at failure, failed requests, console errors
|
|
49
|
+
every run ──▶ analyze-history (flaky / broken / slow / which commit broke it)
|
|
33
50
|
```
|
|
34
51
|
|
|
35
|
-
##
|
|
52
|
+
## Key features
|
|
53
|
+
|
|
54
|
+
- **Fixes are proven, not guessed.** Every fix is re-run before it's kept, and undone if the test is still red.
|
|
55
|
+
- **Honest.** It won't paper over a real bug to make a test pass.
|
|
56
|
+
- **Works on CI failures too.** Load a CI job's report and traces and get the same diagnosis and fixes.
|
|
57
|
+
- **Flakiness with evidence.** Names the flaky test, whether it's a race, shared state or test order, and the seed that reproduces it.
|
|
58
|
+
- **Lean.** Finding and fixing a broken locator took about **5,300 tokens** versus about **7,400**
|
|
59
|
+
with Playwright MCP plus a shell ([benchmark](#benchmark)).
|
|
60
|
+
- **Free and open source.** MIT, no account, no API key, no telemetry, no paid tier.
|
|
61
|
+
|
|
62
|
+
## What you can ask
|
|
36
63
|
|
|
37
64
|
| | You ask your AI… | It does this |
|
|
38
65
|
|---|---|---|
|
|
@@ -49,802 +76,247 @@ run-test ──▶ get-failure ──▶ suggest-fix (apply + re-run) ──▶
|
|
|
49
76
|
| 🔐 | "Test the page after login" | Works behind sign-in with saved sessions, headers and click/fill steps, in Chromium, Firefox or WebKit, and can record or replay network traffic from a HAR file. |
|
|
50
77
|
| 🤖 | "Do it in CI" | A [GitHub Action and `ci` command](#ci-github-action-and-ci-command) post the diagnosis, flaky verdict and fix diffs on your PR. |
|
|
51
78
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
- **Fixes are proven, not guessed.** Every fix is re-run before it's kept.
|
|
55
|
-
- **Honest.** It won't paper over a real bug to make a test pass.
|
|
56
|
-
- **Lean.** In the [benchmark](bench/RESULTS.md), finding and fixing a broken locator took about
|
|
57
|
-
**5,300 tokens** here versus about **7,400** with Playwright MCP plus a shell.
|
|
58
|
-
- **Teaches as it works.** The bundled skill carries a guide to writing tests that stay green
|
|
59
|
-
(locators, waiting, auth, test data, mocking, flake patterns, review checklist).
|
|
60
|
-
- **Ready in one line.** `npx -y playwright-e2e-mcp init` sets up Claude Code, Cursor and VS Code.
|
|
61
|
-
|
|
62
|
-
### Free and open source, for good
|
|
79
|
+
## Requirements
|
|
63
80
|
|
|
64
|
-
-
|
|
65
|
-
-
|
|
66
|
-
-
|
|
67
|
-
- **Free dependencies only:** the MCP SDK (Apache-2.0), zod (MIT), axe-core (MPL-2.0) and the Playwright your project already has (Apache-2.0).
|
|
68
|
-
- **Free to host yourself**, too: the HTTP bridge deploys to a free Vercel account (see [Hosted endpoint](#hosted-endpoint-chatgpt--remote-clients)).
|
|
69
|
-
|
|
70
|
-
<details>
|
|
71
|
-
<summary><b>The technical details</b></summary>
|
|
72
|
-
|
|
73
|
-
- **Structured runs.** Stats plus each failure's `file:line`, kind (assertion, timeout,
|
|
74
|
-
dead dev server, syntax error, crash, full disk…) and a concrete fix hint. Failures that
|
|
75
|
-
share one cause are grouped. Long suites run in the background with live progress.
|
|
76
|
-
- **Diagnosis from the trace.** The DOM at the moment of failure, the failed action, the
|
|
77
|
-
4xx/5xx requests and console errors, read from Playwright's own trace.
|
|
78
|
-
- **Verified fixes.** `suggest-fix` matches a broken locator against the DOM at failure, swaps
|
|
79
|
-
in a Playwright locator proven unique on the page
|
|
80
|
-
(`getByRole('button', { name: 'Proceed to checkout' })`), re-runs the test and reverts the
|
|
81
|
-
edit if it is still red. When nothing on the page plausibly matches, it says the element is
|
|
82
|
-
missing and does not patch the test.
|
|
83
|
-
- **Flakiness with evidence.** `diagnose-flaky` repeats failing tests in one process and
|
|
84
|
-
gives a verdict per test; `isolate: true` re-runs them alone to tell a shared-state bug from
|
|
85
|
-
a race inside the test, and `order: true` runs files in random orders to catch an order
|
|
86
|
-
dependency, naming the test and the seed that reproduces it. `analyze-history` separates
|
|
87
|
-
flaky, regressed and broken tests from the local run record.
|
|
88
|
-
- **Accessibility.** `audit-accessibility` runs axe-core against WCAG 2.2 AA on any page,
|
|
89
|
-
signed in or after steps, with offending elements as verified locators.
|
|
90
|
-
- **Performance.** `audit-performance` measures lab Web Vitals (LCP, CLS, TBT, FCP, TTFB, and
|
|
91
|
-
INP for the interactions in `actions`), optionally as a throttled phone and as the median of
|
|
92
|
-
several runs, rates them against Google's thresholds or your budgets, and names the LCP
|
|
93
|
-
element and the elements that shifted as verified locators.
|
|
94
|
-
- **Ambiguous locators.** When a locator suddenly matches two elements (strict mode
|
|
95
|
-
violation), `suggest-fix` takes the unique locator Playwright reports for the intended
|
|
96
|
-
match: the exact-name one, or the one you pick.
|
|
97
|
-
- **Pages behind a login, in any state.** Every page tool accepts a Playwright
|
|
98
|
-
`storageState`, extra headers, click/fill steps to reach a modal or a later step, and
|
|
99
|
-
network mocks to force error, empty and slow states.
|
|
100
|
-
|
|
101
|
-
</details>
|
|
81
|
+
- Node.js 20 or newer
|
|
82
|
+
- A project that uses `@playwright/test`, with a browser installed (`npx playwright install chromium`)
|
|
83
|
+
- Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Codex, Gemini CLI or any other MCP client
|
|
102
84
|
|
|
103
|
-
|
|
85
|
+
## Getting started
|
|
104
86
|
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
## Demo
|
|
110
|
-
|
|
111
|
-
**Hosted endpoint** — what an `initialize` + `tools/list` round-trip against
|
|
112
|
-
`https://playwright-e2e-mcp.vercel.app/api/mcp` returns for a client that sends the
|
|
113
|
-
bearer token (without one, only `list-tests` and `get-failure` are listed):
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
**A real test run** — `run-test` served over stdio by `npx -y playwright-e2e-mcp`
|
|
118
|
-
against the bundled `examples/sample-test.spec.ts` (actual output, unedited):
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
Images are rendered with `node scripts/gen-demo-images.mjs`: the run-test card is real
|
|
123
|
-
captured output; the endpoint card is an illustration of the authenticated listing.
|
|
124
|
-
|
|
125
|
-
> If this saves you a debugging session, a ⭐ on the repo helps other Playwright users find it.
|
|
126
|
-
|
|
127
|
-
## Install
|
|
128
|
-
|
|
129
|
-
Works with Claude Desktop, Claude Code, Cursor, Windsurf, Codex, Gemini CLI, Freebuff
|
|
130
|
-
and every other MCP client — pick whichever route fits:
|
|
131
|
-
|
|
132
|
-
**Let your agent do it.** Paste this into Claude Code, Cursor, Codex, Gemini CLI or any
|
|
133
|
-
other coding agent:
|
|
134
|
-
|
|
135
|
-
> Install the playwright-e2e-mcp server and skill by following
|
|
136
|
-
> https://raw.githubusercontent.com/xsignon-secure/E2E/main/llms-install.md
|
|
137
|
-
|
|
138
|
-
or run the installer yourself from the project root. It merges the server into each
|
|
139
|
-
client's project config it finds (Claude Code, Cursor, VS Code, Gemini CLI, Codex) and
|
|
140
|
-
installs the agent skill, leaving your other servers alone:
|
|
87
|
+
**Fastest:** run this in your project root. It adds the server to every client config it finds
|
|
88
|
+
(Claude Code, Cursor, VS Code, Gemini CLI, Codex) and installs the agent skill, leaving your
|
|
89
|
+
other servers alone:
|
|
141
90
|
|
|
142
91
|
```bash
|
|
143
92
|
npx -y playwright-e2e-mcp init # --client all | cursor,vscode … · --dry-run
|
|
144
93
|
```
|
|
145
94
|
|
|
146
|
-
|
|
147
|
-
| --- | --- |
|
|
148
|
-
| **One command, any project** | `npx -y playwright-e2e-mcp init` |
|
|
149
|
-
| **Claude Code plugin (server + skill)** | `/plugin marketplace add xsignon-secure/E2E` then `/plugin install playwright-e2e-mcp@xsignon-secure` |
|
|
150
|
-
| **Gemini CLI extension (server + skill)** | `gemini extensions install https://github.com/xsignon-secure/E2E` |
|
|
151
|
-
| **Skill only, any agent** | `npx skills add xsignon-secure/E2E` |
|
|
152
|
-
| **npm (canonical, fastest)** | `npx -y playwright-e2e-mcp` |
|
|
153
|
-
| **MCP Registry** (registry-aware clients discover it automatically) | `io.github.xsignon-secure/E2E` — [listing](https://registry.modelcontextprotocol.io/) |
|
|
154
|
-
| **Any client, no npm account needed** | `npx -y github:xsignon-secure/E2E#v0.5.0` (pin a release tag) |
|
|
155
|
-
| **Claude Desktop, zero Node setup** | double-click the [`.mcpb` extension](https://github.com/xsignon-secure/E2E/releases) |
|
|
156
|
-
| **Remote-only clients (ChatGPT connectors)** | `https://playwright-e2e-mcp.vercel.app/api/mcp` |
|
|
157
|
-
|
|
158
|
-
One line for the common clients:
|
|
159
|
-
|
|
160
|
-
```bash
|
|
161
|
-
claude mcp add playwright-e2e -- npx -y playwright-e2e-mcp
|
|
162
|
-
code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","playwright-e2e-mcp"]}'
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
Only need the fix loop? `--tools=core` (or `PW_MCP_TOOLS=core`) exposes six tools and
|
|
166
|
-
cuts the tool-definition context by about three quarters; see [Configuration](#configuration).
|
|
167
|
-
|
|
168
|
-
Details and per-client config: [Installation](#installation) ·
|
|
169
|
-
[MCP client configuration](#mcp-client-configuration).
|
|
170
|
-
|
|
171
|
-
---
|
|
172
|
-
|
|
173
|
-
## Tools
|
|
174
|
-
|
|
175
|
-
| Tool | Purpose |
|
|
176
|
-
| --- | --- |
|
|
177
|
-
| `run-test` | Run Playwright tests and return stats, failures, diagnostics and hints; reports progress, and `background: true` returns a run id for long suites |
|
|
178
|
-
| `get-run-status` | Progress or final result of a background run; can wait for it or cancel it |
|
|
179
|
-
| `get-failure` | Deep analysis of one failure: stack, expected/actual, **DOM snapshot at failure (from the Playwright trace)**, failed requests, console errors, next steps |
|
|
180
|
-
| `load-report` | Load a run from CI: a JSON report or a folder of sharded blob reports; CI paths are mapped onto the checkout and traces found in the unpacked artifact, so `get-failure`, `suggest-fix` and `diagnose-flaky` work on it, and the run joins the history with its commit |
|
|
181
|
-
| `suggest-fix` | Patch a failure: a drifted locator is replaced with one **proven unique** on the DOM at failure, an ambiguous one (strict mode violation) gets the unique locator of the intended match, changed copy gets the new expected text; `apply: true` writes it, re-runs the test and reverts if still red |
|
|
182
|
-
| `inspect-page` | Open a URL headlessly (optionally signed in, after steps) and return elements with **verified Playwright locators**, CSS selectors, visibility, text and console output, or the **accessibility snapshot with a ready `toMatchAriaSnapshot` assertion** |
|
|
183
|
-
| `list-tests` | List available tests (`file`, `line`, full title, projects) with filtering |
|
|
184
|
-
| `validate-selector` | Check a CSS selector, Playwright selector or **locator expression** against a live page: validity, match count, and a sturdier verified locator for each match |
|
|
185
|
-
| `generate-e2e-test` | Scaffold a Playwright test from a description using the project's **real** selectors and verified locators from the live page, or **walk a flow** step by step on the live page and write a spec that is run once to prove it passes |
|
|
186
|
-
| `compare-visual-state` | Visual regression: screenshot before/after a change and report *what* moved and how colors shifted |
|
|
187
|
-
| `audit-accessibility` | Run axe-core (WCAG 2.2 AA by default) on a page, signed in or after steps, and return violations worst first with **verified locators** for the offending elements and a regression check |
|
|
188
|
-
| `audit-performance` | Measure lab **Web Vitals** (LCP, CLS, TBT, FCP, TTFB, INP during steps), optionally as a throttled phone and the median of several runs; rate them or check budgets, name the LCP and shifting elements as verified locators, list heavy and render-blocking requests, and return a budget test |
|
|
189
|
-
| `diagnose-flaky` | Run a failing test 2–10 times **with retries disabled** and return an evidence verdict: `CONSISTENTLY FAILING`, `FLAKY` or `NOT REPRODUCING`; tells parallelism and **test-order** bugs from races |
|
|
190
|
-
| `analyze-history` | Rank flaky, regressed and broken tests from the local run history, group failures by cause, list the slowest tests, show stability by day, name **the commit where each test went bad**, or show one test's timeline |
|
|
191
|
-
|
|
192
|
-
Two **prompts** package the main workflows for clients that show them as slash commands:
|
|
193
|
-
`fix-failing-test` (optional `test`) and `triage-flaky-tests`.
|
|
95
|
+
**Or let your agent do it.** Paste this into any coding agent:
|
|
194
96
|
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
`inspect-page`, `validate-selector`, `compare-visual-state`, `audit-accessibility` and
|
|
198
|
-
`audit-performance` accept the same page-state arguments (`suggest-fix` accepts
|
|
199
|
-
`storageState` and `headers` for a live `url`):
|
|
97
|
+
> Install the playwright-e2e-mcp server and skill by following
|
|
98
|
+
> https://raw.githubusercontent.com/xsignon-secure/E2E/main/llms-install.md
|
|
200
99
|
|
|
201
|
-
|
|
202
|
-
| --- | --- | --- |
|
|
203
|
-
| `storageState` | string | Path inside the project to a Playwright storageState JSON (cookies + localStorage), e.g. `playwright/.auth/user.json` from your setup project. Defaults to `PW_MCP_STORAGE_STATE` |
|
|
204
|
-
| `headers` | object | Extra HTTP headers for every request, e.g. `{ "Authorization": "Bearer …" }` |
|
|
205
|
-
| `actions` | array | Up to 20 steps run after load: `{ "type": "click" \| "dblclick" \| "hover" \| "fill" \| "press" \| "check" \| "uncheck" \| "select" \| "wait" \| "goto", "locator"?, "value"?, "url"?, "ms"? }`. Locators use Playwright syntax, e.g. `getByRole('button', { name: 'Open settings' })` |
|
|
206
|
-
| `routes` | array | Up to 20 network mocks applied before the page loads, first match wins: `{ "url": "**/api/cart", "method"?, "status"?, "json"? \| "body"?, "contentType"?, "headers"?, "abort"?, "delayMs"? }`. `json`, `body` or `status` fulfils the request, `abort: true` fails it, and `delayMs` alone slows the real request |
|
|
207
|
-
| `viewport` | object | `{ "width": 390, "height": 844 }` (default 1280×720) |
|
|
208
|
-
| `har` | object | `{ "path": "tests/fixtures/cart.har", "mode"?: "replay" \| "record", "url"?: "**/api/**", "notFound"?: "abort" \| "fallback" }`. `replay` (default) answers matching requests from the file, `record` saves the page's traffic to it. The file and its folder must be inside the project. `routes` mocks still win over the HAR |
|
|
209
|
-
| `browser` | `chromium` \| `firefox` \| `webkit` | Browser engine (default `chromium`; the engine must be installed with `npx playwright install`). `audit-performance` is Chromium-only, and WebKit is refused while private addresses are blocked |
|
|
210
|
-
|
|
211
|
-
Mocks make states that are hard to reach easy to inspect, audit or measure: the error
|
|
212
|
-
banner after a 500, the empty list, the spinner of a slow API, the page without a
|
|
213
|
-
third-party script. The response says how many requests each mock handled and flags a mock
|
|
214
|
-
that matched nothing, which usually means a wrong glob:
|
|
100
|
+
**Standard config** works in most clients:
|
|
215
101
|
|
|
216
102
|
```json
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
]
|
|
103
|
+
{
|
|
104
|
+
"mcpServers": {
|
|
105
|
+
"playwright-e2e": {
|
|
106
|
+
"command": "npx",
|
|
107
|
+
"args": ["-y", "playwright-e2e-mcp"]
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
}
|
|
222
111
|
```
|
|
223
112
|
|
|
224
|
-
|
|
225
|
-
(
|
|
226
|
-
The same file works in a spec with `page.routeFromHAR()`.
|
|
113
|
+
The server uses its working directory as the project root. If your client starts it somewhere
|
|
114
|
+
else (such as your home directory), add `"env": { "PW_MCP_PROJECT_ROOT": "/path/to/your/project" }`.
|
|
227
115
|
|
|
228
|
-
|
|
229
|
-
|
|
116
|
+
<details>
|
|
117
|
+
<summary><b>Claude Code</b></summary>
|
|
230
118
|
|
|
231
|
-
|
|
119
|
+
Server and skill, as a plugin:
|
|
232
120
|
|
|
233
|
-
| Argument | Type | Description |
|
|
234
|
-
| --- | --- | --- |
|
|
235
|
-
| `projectRoot` | string | Project directory inside the configured root (default: server working directory) |
|
|
236
|
-
| `testFiles` | string[] | Files/directories relative to the root; `file:line` supported. Omit to run everything |
|
|
237
|
-
| `grep` | string | Only run tests whose title matches this regex |
|
|
238
|
-
| `browser` | `chromium` \| `firefox` \| `webkit` | Playwright project to run (matched against config project names) |
|
|
239
|
-
| `headed` | boolean | Visible browser window |
|
|
240
|
-
| `timeoutMs` | number | Hard wall-clock limit for the run (default `120000`); the whole process tree is killed past it and **partial results are returned** |
|
|
241
|
-
| `testTimeoutMs` | number | Per-test timeout passed to Playwright |
|
|
242
|
-
| `workers` / `retries` | number | Passed through to Playwright |
|
|
243
|
-
| `config` | string | `playwright.config` path **or 1-based index** when the project has several |
|
|
244
|
-
| `retryOnFailure` | boolean | Auto-retry failures **once** before reporting them (default `true`; ignored when `retries` is set) |
|
|
245
|
-
| `lastFailed` | boolean | Only re-run tests that failed in the previous run (Playwright `--last-failed`) — the fast fix → re-run loop |
|
|
246
|
-
| `args` | string[] | Extra Playwright flags from an allowlist (`--repeat-each=N`, `--max-failures=N`, `--update-snapshots`, `--shard=1/3`, `--trace=on`, …); values go after `=` and are checked, and flags that take a path, such as `--config` or `--output`, are rejected |
|
|
247
|
-
| `background` | boolean | Return a run id at once; poll `get-run-status` for progress and the result |
|
|
248
|
-
|
|
249
|
-
Progress: clients that send a progress token get a notification as each test finishes
|
|
250
|
-
(`3/12 tests done, 1 failed`). Every run is appended to the local history used by
|
|
251
|
-
`analyze-history`.
|
|
252
|
-
|
|
253
|
-
Flakiness handling: by default the server injects `--retries=1` (unless the config
|
|
254
|
-
already sets `retries`), so a test that passes on the retry is reported as **flaky**,
|
|
255
|
-
not failed. Traces are captured automatically (`--trace=retain-on-failure`) so
|
|
256
|
-
`get-failure` can show the DOM at the moment of failure.
|
|
257
|
-
|
|
258
|
-
Example result:
|
|
259
|
-
|
|
260
|
-
```markdown
|
|
261
|
-
## Playwright run — ❌ FAILED
|
|
262
|
-
|
|
263
|
-
**Command:** `playwright test --config playwright.config.ts tests/checkout.spec.ts --reporter=json`
|
|
264
|
-
**duration 4.2s · exit 1 · config `playwright.config.ts`**
|
|
265
|
-
|
|
266
|
-
| passed | failed | flaky | skipped | duration |
|
|
267
|
-
| ---: | ---: | ---: | ---: | ---: |
|
|
268
|
-
| 0 | 1 | 0 | 0 | 1.1s |
|
|
269
|
-
|
|
270
|
-
### ❌ 1 failing test(s)
|
|
271
|
-
|
|
272
|
-
### 1 of 1. checkout.spec.ts › pays with card
|
|
273
|
-
**File:** `checkout.spec.ts:5` | **failed · server-unreachable**
|
|
274
|
-
|
|
275
|
-
### ⚠️ SERVER_NOT_RUNNING
|
|
276
|
-
Your app (dev server) does not appear to be reachable. Start it in another terminal
|
|
277
|
-
(e.g. npm run dev / npm start), keep it running, then retry — or configure `webServer`
|
|
278
|
-
in playwright.config.* so Playwright starts it automatically.
|
|
279
121
|
```
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
| Argument | Type | Description |
|
|
284
|
-
| --- | --- | --- |
|
|
285
|
-
| `index` | number | 1-based failure index from the last run (default `1`) |
|
|
286
|
-
| `projectRoot` | string | Only used when re-reading the stored report |
|
|
287
|
-
|
|
288
|
-
Returns the message/code frame, expected vs actual, stack, failure kind with a
|
|
289
|
-
diagnosis, the test's console output, **the DOM snapshot from the Playwright trace
|
|
290
|
-
(plus the failed action, its selector, and the action log leading up to it)**,
|
|
291
|
-
**the network requests that failed** (4xx/5xx, dead endpoints, no-response — with
|
|
292
|
-
method, URL, status and resource type), **the console errors/warnings the page
|
|
293
|
-
logged before the failure**, and
|
|
294
|
-
numbered next steps (re-run this single test by `file:line`, headed/debug mode,
|
|
295
|
-
`validate-selector` when the message mentions a locator, …).
|
|
296
|
-
|
|
297
|
-
### `inspect-page`
|
|
298
|
-
|
|
299
|
-
| Argument | Type | Description |
|
|
300
|
-
| --- | --- | --- |
|
|
301
|
-
| `url` | string | Full http(s) URL to open (required) |
|
|
302
|
-
| `view` | `elements` \| `locators` \| `aria` | `locators` returns a compact map of interactive elements with verified locators (cheapest); `aria` the accessibility snapshot of the page (or of `selector`) with a ready `toMatchAriaSnapshot` assertion; `elements` (default) the DOM inventory |
|
|
303
|
-
| `projectRoot` | string | Project whose Playwright launches the browser |
|
|
304
|
-
| `selector` | string | Inspect matches of this selector instead of the whole DOM |
|
|
305
|
-
| `waitFor` | string | Wait for a selector (CSS or `text=…`) before inspecting |
|
|
306
|
-
| `waitUntil` | `load` \| `domcontentloaded` \| `networkidle` | Navigation wait condition |
|
|
307
|
-
| `includeHtml` | boolean | Include the rendered HTML (capped) |
|
|
308
|
-
| `maxHtmlChars` | number | HTML cap, default `20000` |
|
|
309
|
-
| `timeoutMs` | number | Overall limit, default `45000` |
|
|
310
|
-
| `storageState` / `headers` / `actions` / `routes` / `har` / `browser` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
|
|
311
|
-
|
|
312
|
-
Each element comes with Playwright locators that were **checked on the page** to resolve to
|
|
313
|
-
exactly that element, best first (role + accessible name, test id, label, placeholder, text,
|
|
314
|
-
then CSS), plus its unique CSS selector, visibility, box and text. Console messages are
|
|
315
|
-
captured, errors first. A locator map looks like:
|
|
316
|
-
|
|
317
|
-
```markdown
|
|
318
|
-
1. `getByRole('button', { name: 'Proceed to checkout' })` — button "Proceed to checkout" · also `getByTestId('checkout-btn')`
|
|
319
|
-
2. `getByLabel('Search')` — searchbox "Search"
|
|
122
|
+
/plugin marketplace add xsignon-secure/E2E
|
|
123
|
+
/plugin install playwright-e2e-mcp@xsignon-secure
|
|
320
124
|
```
|
|
321
125
|
|
|
322
|
-
|
|
323
|
-
locks it in a test:
|
|
126
|
+
Or the server only:
|
|
324
127
|
|
|
325
|
-
```
|
|
326
|
-
|
|
327
|
-
- list:
|
|
328
|
-
- listitem: Real item
|
|
329
|
-
`);
|
|
128
|
+
```bash
|
|
129
|
+
claude mcp add playwright-e2e -- npx -y playwright-e2e-mcp
|
|
330
130
|
```
|
|
331
131
|
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
| Argument | Type | Description |
|
|
335
|
-
| --- | --- | --- |
|
|
336
|
-
| `projectRoot` | string | Project directory |
|
|
337
|
-
| `config` | string | Config path or 1-based index |
|
|
338
|
-
| `testDir` | string | Restrict scanning to a directory (must stay inside the project) |
|
|
339
|
-
| `filter` | string | Case-insensitive substring filter on `file › title` |
|
|
340
|
-
| `limit` | number | Max tests returned, default `500` |
|
|
341
|
-
|
|
342
|
-
Uses `playwright test --list` when Playwright works, and **falls back to a source scan**
|
|
343
|
-
(keeping the reason) when the install or a spec file is broken.
|
|
344
|
-
|
|
345
|
-
### `validate-selector`
|
|
132
|
+
</details>
|
|
346
133
|
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
| `url` | string | Live page to test against (required) |
|
|
350
|
-
| `selector` | string | CSS, a Playwright selector (`text=`, `role=`, `>>`) or a locator expression such as `getByRole('button', { name: 'Save' }).first()` (required) |
|
|
351
|
-
| `projectRoot` | string | Project whose Playwright launches the browser |
|
|
352
|
-
| `timeoutMs` | number | Overall limit, default `45000` |
|
|
353
|
-
| `storageState` / `headers` / `actions` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
|
|
134
|
+
<details>
|
|
135
|
+
<summary><b>Claude Desktop</b></summary>
|
|
354
136
|
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
137
|
+
Download the `.mcpb` extension from the [latest release](https://github.com/xsignon-secure/E2E/releases/latest)
|
|
138
|
+
and double-click it. It ships its own dependencies, so no Node setup is needed. On install it
|
|
139
|
+
asks for your **project root**: choose the project folder, not your home directory.
|
|
358
140
|
|
|
359
|
-
|
|
360
|
-
a strict-mode warning when more than one element matches, a hint when a CSS selector is
|
|
361
|
-
brittle (`:nth-child`, long descendant chains, XPath), `✅ VALID — 0 matches` with debugging
|
|
362
|
-
advice, or `❌ INVALID` with the parse error.
|
|
141
|
+
You can also add the standard config to `claude_desktop_config.json`.
|
|
363
142
|
|
|
364
|
-
|
|
143
|
+
</details>
|
|
365
144
|
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
| `description` | string | What the test should cover (required) |
|
|
369
|
-
| `pageUrl` | string | Page the test starts on (default: `baseURL` / `webServer.url` from config) |
|
|
370
|
-
| `testDir` / `file` | string | Where to write the spec (default: detected `testDir` + `generated/<slug>.spec.ts`); `file` must end in `.spec.*` or `.test.*` |
|
|
371
|
-
| `write` | boolean | Write the file to disk (default `true`) |
|
|
372
|
-
| `overwrite` | boolean | Replace an existing spec at the target path; only specs this tool generated can be replaced |
|
|
373
|
-
| `liveInspect` | boolean | Cross-check selectors against the live page (default on when a URL is known) |
|
|
374
|
-
| `steps` | array | **Walk the flow**: up to 30 steps in the [`actions`](#page-state-signed-in-pages-headers-and-steps) format, run on the live page before anything is written |
|
|
375
|
-
| `assertScope` | string | With `steps`: CSS scope of the end-state accessibility snapshot (default `body`) |
|
|
376
|
-
| `storageState` | string | With `steps`: signed-in session for the walk; the spec uses it too (`test.use`) |
|
|
377
|
-
| `verify` / `project` / `timeoutMs` | — | With `steps`: run the written spec once (default `true`), in this Playwright project |
|
|
378
|
-
| `projectRoot` / `config` | string | As with the other tools |
|
|
379
|
-
|
|
380
|
-
**With `steps`**, the tool walks the flow on the live page first. If a step's locator does
|
|
381
|
-
not work, it stops there, writes nothing and says which step failed. Otherwise it writes a
|
|
382
|
-
spec that replays the steps, asserts the URL path and the accessibility snapshot where the
|
|
383
|
-
flow landed (headings only when the snapshot is long), and runs it once:
|
|
384
|
-
|
|
385
|
-
```ts
|
|
386
|
-
test('rename in settings', async ({ page }) => {
|
|
387
|
-
await page.goto('http://localhost:3000/account');
|
|
388
|
-
await page.getByRole('button', { name: 'Open settings' }).click();
|
|
389
|
-
await page.getByLabel('Nickname').fill('Ada');
|
|
390
|
-
|
|
391
|
-
// End state
|
|
392
|
-
await expect(page.locator("dialog")).toMatchAriaSnapshot(`
|
|
393
|
-
- dialog:
|
|
394
|
-
- heading "Settings" [level=2]
|
|
395
|
-
- text: Nickname
|
|
396
|
-
- textbox "Nickname": Ada
|
|
397
|
-
- button "Save settings"
|
|
398
|
-
`);
|
|
399
|
-
});
|
|
400
|
-
```
|
|
145
|
+
<details>
|
|
146
|
+
<summary><b>Cursor</b></summary>
|
|
401
147
|
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
ranks verified-live selectors first, writes a spec built from them, and reports each
|
|
406
|
-
selector with its source `file:line`. When the source has no usable selectors, it uses the
|
|
407
|
-
verified locators from the live page that best match the description.
|
|
148
|
+
Click [Install in Cursor](https://cursor.com/en/install-mcp?name=playwright-e2e&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInBsYXl3cmlnaHQtZTJlLW1jcCJdfQ==),
|
|
149
|
+
or go to **Settings → MCP → Add new MCP server** and paste the standard config. For one
|
|
150
|
+
project only, put it in `.cursor/mcp.json`.
|
|
408
151
|
|
|
409
|
-
|
|
152
|
+
</details>
|
|
410
153
|
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
| `url` | string | Page to capture (required) |
|
|
414
|
-
| `name` | string | Baseline id, e.g. `checkout-page` (letters, digits, `. _ -`) |
|
|
415
|
-
| `action` | `compare` \| `baseline` | `compare` (default) diffs; `baseline` re-captures the reference |
|
|
416
|
-
| `selector` | string | Capture just this element |
|
|
417
|
-
| `fullPage` | boolean | Capture the full scrollable page |
|
|
418
|
-
| `tolerance` | number | Percent of pixels that may differ (default `0.1`) |
|
|
419
|
-
| `pixelThreshold` | number | Per-pixel channel delta considered different (default `60`) |
|
|
420
|
-
| `waitUntil` / `waitFor` / `timeoutMs` | — | As with `inspect-page` |
|
|
421
|
-
| `storageState` / `headers` / `actions` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
|
|
422
|
-
|
|
423
|
-
The first call saves a baseline under `.pw-mcp/visual/` (add that to `.gitignore`, or
|
|
424
|
-
commit it for CI comparisons). Later calls report changed-pixel counts, **merged
|
|
425
|
-
regions** (`(x, y) 120×40 — 1,200 px`), the **average color shift** ("blue → red"),
|
|
426
|
-
and write a red-highlighted diff image for review.
|
|
427
|
-
|
|
428
|
-
### `diagnose-flaky`
|
|
429
|
-
|
|
430
|
-
| Argument | Type | Description |
|
|
431
|
-
| --- | --- | --- |
|
|
432
|
-
| `testFiles` | string[] | Tests to diagnose (`file:line` supported). Defaults to the tests that failed in the most recent run |
|
|
433
|
-
| `runs` | number | Repetitions, `2–10` (default `3`) |
|
|
434
|
-
| `isolate` | boolean | Also tell parallelism bugs from flakiness: repeat fully parallel (`workers` default `4`), then repeat the failing tests with one worker |
|
|
435
|
-
| `order` | boolean | Also run the target files in `runs` random orders with one worker (Playwright 1.64+ `--shuffle`), replay a failing order, and name the test it relies on or is broken by |
|
|
436
|
-
| `separateProcesses` | boolean | Start a fresh Playwright process per repetition instead of one `--repeat-each` process (slower; for flakes that depend on global setup or a cold start) |
|
|
437
|
-
| `browser` / `headed` / `workers` / `config` | — | As with `run-test` |
|
|
438
|
-
| `timeoutMs` | number | Hard wall-clock limit **per repetition** (default `120000`); one `--repeat-each` process gets this × `runs` |
|
|
439
|
-
| `projectRoot` | string | Project directory |
|
|
440
|
-
|
|
441
|
-
All repetitions run in **one Playwright process** with `--repeat-each` and
|
|
442
|
-
`--retries=0`, so each attempt is honest evidence and the browser and workers
|
|
443
|
-
start once instead of once per run (about 25% faster on the integration fixture,
|
|
444
|
-
more for suites with expensive setup). A small extra reporter records every
|
|
445
|
-
attempt, because Playwright's JSON report keeps only one result per test.
|
|
446
|
-
|
|
447
|
-
The verdict is **per test**: a table of passes, failures, duration spread and
|
|
448
|
-
first error for each test, plus the verdict block for the worst one:
|
|
449
|
-
|
|
450
|
-
- **❌ CONSISTENTLY FAILING** — failed every run (same error → reproducible bug,
|
|
451
|
-
different errors → still broken, just noisy). Fix it; it is not flaky.
|
|
452
|
-
- **⚠️ FLAKY** — some runs passed. Includes `N of M` counts and whether the
|
|
453
|
-
failures share one signature (real intermittent bug) or vary (timing/environment
|
|
454
|
-
instability).
|
|
455
|
-
- **✅ NOT REPRODUCING** — passed every re-run; the original failure was one-off.
|
|
456
|
-
|
|
457
|
-
With `isolate: true` a second table compares each failing test **in parallel**
|
|
458
|
-
and **alone**:
|
|
459
|
-
|
|
460
|
-
| Reading | Meaning |
|
|
461
|
-
| --- | --- |
|
|
462
|
-
| 🔀 fails only in parallel | Passes every run alone: shared state between tests (same account, DB rows, files, ports). Give each worker its own data or run the file serially |
|
|
463
|
-
| ⏱️ flaky even alone | The race is inside the test or the app: missing `await`, asserting before the UI settles |
|
|
464
|
-
| ❌ fails alone too | Not a parallelism problem; a real bug |
|
|
154
|
+
<details>
|
|
155
|
+
<summary><b>VS Code / GitHub Copilot</b></summary>
|
|
465
156
|
|
|
466
|
-
|
|
467
|
-
lines) in random orders with one worker and `--fully-parallel`, so every test is
|
|
468
|
-
scheduled on its own. A test that fails in some orders is replayed in a failing
|
|
469
|
-
order; if it fails again it is read as **🔗 order dependent**, otherwise as
|
|
470
|
-
**flaky regardless of order**:
|
|
157
|
+
Click the **Install in VS Code** button above, or run:
|
|
471
158
|
|
|
159
|
+
```bash
|
|
160
|
+
code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","playwright-e2e-mcp"]}'
|
|
472
161
|
```
|
|
473
|
-
| `tests/cart.spec.ts:10` checkout lists the book | 2/3 | `4` | 🔗 order dependent … |
|
|
474
|
-
|
|
475
|
-
**checkout lists the book**
|
|
476
|
-
- Relies on: `adds a book to the cart` (ran before it in every passing order, after it in a failing one)
|
|
477
|
-
- Reproduce: `npx playwright test tests/cart.spec.ts --workers=1 --fully-parallel --retries=0 --shuffle=4`
|
|
478
|
-
```
|
|
479
|
-
|
|
480
|
-
This is the failure that otherwise reads as "passes alone, fails in the suite".
|
|
481
|
-
Set `PW_MCP_SHUFFLE_SEEDS=4,7` on the server to replay given orders, for
|
|
482
|
-
example to check a fix.
|
|
483
|
-
|
|
484
|
-
Each repetition is recorded in the run history, and the last failing run is
|
|
485
|
-
stored, so `analyze-history` and `get-failure` work right afterwards.
|
|
486
162
|
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
| Argument | Type | Description |
|
|
490
|
-
| --- | --- | --- |
|
|
491
|
-
| `url` | string | Page to audit (required) |
|
|
492
|
-
| `standard` | `wcag2a` \| `wcag2aa` \| `wcag21aa` \| `wcag22aa` \| `best-practice` \| `all` | Rule set (default `wcag22aa`: WCAG 2.0–2.2 level A and AA; `best-practice` adds axe's extra rules; `all` runs every rule) |
|
|
493
|
-
| `minImpact` | `critical` \| `serious` \| `moderate` \| `minor` | Hide violations below this impact |
|
|
494
|
-
| `selector` | string | Audit only this part of the page |
|
|
495
|
-
| `waitFor` / `waitUntil` / `timeoutMs` | — | As with `inspect-page` (`waitUntil` default `load`, `timeoutMs` default `60000`) |
|
|
496
|
-
| `storageState` / `headers` / `actions` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps) |
|
|
163
|
+
</details>
|
|
497
164
|
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
Example output:
|
|
165
|
+
<details>
|
|
166
|
+
<summary><b>Codex</b></summary>
|
|
501
167
|
|
|
168
|
+
```bash
|
|
169
|
+
codex mcp add playwright-e2e -- npx -y playwright-e2e-mcp
|
|
502
170
|
```
|
|
503
|
-
## ♿ 3 accessibility violations (2 critical, 1 serious)
|
|
504
171
|
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
- `getByTestId('go')`: Element does not have inner text that is visible to screen readers
|
|
172
|
+
A Playwright run with retries can take longer than Codex's default 60 s tool timeout. Raise
|
|
173
|
+
the timeouts in `~/.codex/config.toml`:
|
|
508
174
|
|
|
509
|
-
|
|
510
|
-
-
|
|
175
|
+
```toml
|
|
176
|
+
[mcp_servers.playwright-e2e]
|
|
177
|
+
command = "npx"
|
|
178
|
+
args = ["-y", "playwright-e2e-mcp"]
|
|
179
|
+
startup_timeout_sec = 60
|
|
180
|
+
tool_timeout_sec = 600
|
|
511
181
|
```
|
|
512
182
|
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
### `audit-performance`
|
|
183
|
+
</details>
|
|
516
184
|
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
| `url` | string | Page to measure (required) |
|
|
520
|
-
| `throttle` | `none` \| `mobile` | `mobile` is Lighthouse's phone preset: 412×823 touch viewport, 4x CPU slowdown, slow 4G. Default `none` (desktop, unthrottled) |
|
|
521
|
-
| `cpuSlowdown` | number | Override the CPU slowdown factor (1–20) |
|
|
522
|
-
| `runs` | number | Load the page 1–5 times and report the median (default `1`); the LCP range is shown |
|
|
523
|
-
| `budgets` | object | `{ lcpMs, cls, inpMs, tbtMs, fcpMs, ttfbMs, transferKb, requests }`, any subset. With budgets the verdict is pass/fail; without, metrics are rated against Google's thresholds |
|
|
524
|
-
| `settleMs` | number | Keep observing after load for late images and shifts (default `1500`) |
|
|
525
|
-
| `waitFor` / `waitUntil` / `timeoutMs` | — | As with `inspect-page` (`waitUntil` default `load`, `timeoutMs` per run, default `60000`) |
|
|
526
|
-
| `storageState` / `headers` / `actions` / `routes` / `viewport` | — | [Page state](#page-state-signed-in-pages-headers-and-steps). Clicks and key presses in `actions` are measured as interactions (INP) |
|
|
527
|
-
|
|
528
|
-
The metrics come from the browser's own performance entries (the same ones Lighthouse and
|
|
529
|
-
the `web-vitals` library read) and the bytes from the DevTools protocol, so cross-origin
|
|
530
|
-
sizes count too. Example output:
|
|
185
|
+
<details>
|
|
186
|
+
<summary><b>Gemini CLI / Qwen Code</b></summary>
|
|
531
187
|
|
|
532
|
-
|
|
533
|
-
## ⚡ 3 of 6 metrics need work (CLS, INP, TBT)
|
|
188
|
+
Server and skill, as an extension:
|
|
534
189
|
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
| LCP · largest contentful paint | 412 ms | 🟢 good |
|
|
538
|
-
| CLS · cumulative layout shift | 0.109 | 🟠 needs improvement |
|
|
539
|
-
| INP · slowest interaction (lab) | 360 ms | 🟠 needs improvement |
|
|
540
|
-
| TBT · total blocking time | 306 ms | 🟠 needs improvement |
|
|
541
|
-
|
|
542
|
-
### What to fix
|
|
543
|
-
- **CLS 0.109**: these moved: `getByRole('img', { name: 'Summer sale' })`, `getByRole('button', { name: 'Add to cart' })`. Reserve their space (…)
|
|
544
|
-
- **TBT 306 ms** from 2 long task(s), the longest 404 ms: split or defer JavaScript.
|
|
545
|
-
- **INP 360 ms** on pointerdown on button: the handler blocks the next paint; yield (…)
|
|
546
|
-
|
|
547
|
-
### Network
|
|
548
|
-
**Transfer:** 302 kB in 4 requests · DOM: 13 nodes · JS heap: 10.0 MB
|
|
549
|
-
**Render-blocking:**
|
|
550
|
-
- stylesheet `http://localhost:4173/style.css` · 324 ms
|
|
190
|
+
```bash
|
|
191
|
+
gemini extensions install https://github.com/xsignon-secure/E2E
|
|
551
192
|
```
|
|
552
193
|
|
|
553
|
-
|
|
554
|
-
numbers from one headless Chromium; field data (CrUX, RUM) can differ.
|
|
555
|
-
|
|
556
|
-
### `suggest-fix`
|
|
557
|
-
|
|
558
|
-
| Argument | Type | Description |
|
|
559
|
-
| --- | --- | --- |
|
|
560
|
-
| `index` | number | 1-based failure index from the last run (default `1`) |
|
|
561
|
-
| `apply` | boolean | Write the fix into the spec (default `false`: return the diff only) |
|
|
562
|
-
| `verify` | boolean | After applying, re-run the test and revert the file if it still fails (default `true`) |
|
|
563
|
-
| `allowExpectationUpdate` | boolean | Allow applying a changed expected text/value (default `false`, since it can hide a real regression) |
|
|
564
|
-
| `candidate` | number | For a strict mode violation, which of the listed matches the test meant (1-based) |
|
|
565
|
-
| `url` | string | Heal against this live page instead of the DOM snapshot in the failure trace |
|
|
566
|
-
| `storageState` / `headers` | — | For `url`, as in [Page state](#page-state-signed-in-pages-headers-and-steps) |
|
|
567
|
-
| `config` / `timeoutMs` / `projectRoot` | — | As with `run-test` |
|
|
568
|
-
|
|
569
|
-
Three kinds of fix:
|
|
570
|
-
|
|
571
|
-
- **Ambiguous locator.** A strict mode violation ("resolved to 2 elements") lists each match
|
|
572
|
-
with the unique locator Playwright computed for it. When the broken locator matched by
|
|
573
|
-
substring (`name: 'Save'` hitting *Save* and *Save draft*), the exact-name match is picked
|
|
574
|
-
with `high` confidence. When the matches are equal (the same *Home* link in the header and
|
|
575
|
-
the footer), they are listed and you choose with `candidate`.
|
|
576
|
-
- **Locator drift.** The broken locator is read from the error, its words are matched
|
|
577
|
-
(with synonyms such as *btn → button*, *checkout → proceed to checkout*) against every
|
|
578
|
-
element in the DOM at failure, and the best match's verified-unique locator replaces it.
|
|
579
|
-
Confidence is `high` when the match is clear, `medium` otherwise.
|
|
580
|
-
- **Changed copy.** For `toHaveText`, `toContainText`, `toHaveValue`, `toHaveTitle`,
|
|
581
|
-
`toHaveURL` and `toHaveAttribute`, it proposes the value the page actually showed. It is
|
|
582
|
-
only applied with `allowExpectationUpdate: true`.
|
|
583
|
-
|
|
584
|
-
The response is a unified diff. With `apply: true` the file is written only if it has not
|
|
585
|
-
changed since the failure, the single test is re-run (same project, no retries), and the
|
|
586
|
-
edit is reverted when it is still red. If nothing on the page shares the locator's
|
|
587
|
-
distinctive words, it refuses and reports that the element is most likely **missing**.
|
|
588
|
-
|
|
589
|
-
### `get-run-status`
|
|
590
|
-
|
|
591
|
-
| Argument | Type | Description |
|
|
592
|
-
| --- | --- | --- |
|
|
593
|
-
| `runId` | string | Run id from `run-test` with `background: true` (default: the most recent run) |
|
|
594
|
-
| `waitSeconds` | number | Wait up to this long (max 55) for the run to finish |
|
|
595
|
-
| `cancel` | boolean | Stop the run and kill its browsers |
|
|
596
|
-
|
|
597
|
-
While running it reports tests done / total and failures so far; once finished it returns
|
|
598
|
-
the full `run-test` result, which also becomes the last run for `get-failure` and
|
|
599
|
-
`suggest-fix`. The 20 most recent runs are kept in memory.
|
|
600
|
-
|
|
601
|
-
### `analyze-history`
|
|
602
|
-
|
|
603
|
-
| Argument | Type | Description |
|
|
604
|
-
| --- | --- | --- |
|
|
605
|
-
| `view` | `flaky` \| `patterns` \| `slow` \| `trend` \| `commits` \| `test` | `flaky` (default) ranks unstable tests; `patterns` groups failures by kind and normalized error; `slow` lists tests by median duration and flags ones that got slower; `trend` shows failed and flaky executions per day; `commits` names the first bad commit of each failing or flaky test; `test` shows one test's timeline |
|
|
606
|
-
| `test` | string | Substring of a test's file or title (filters every view) |
|
|
607
|
-
| `lastRuns` | number | How many recent runs to analyze (default `30`) |
|
|
608
|
-
| `limit` | number | Max rows (default `15`) |
|
|
609
|
-
|
|
610
|
-
Every `run-test`, `diagnose-flaky`, `suggest-fix` and `load-report` run appends a line to
|
|
611
|
-
`<project>/.playwright-e2e-mcp/history.jsonl` (the folder carries its own `.gitignore`;
|
|
612
|
-
nothing leaves the machine; `PW_MCP_HISTORY=0` turns it off). The flaky view separates
|
|
613
|
-
tests that flip between pass and fail, tests that started failing and stayed red
|
|
614
|
-
(regressed), and tests that failed every run (broken), each with a `PFFP~` timeline.
|
|
615
|
-
|
|
616
|
-
Each run records the commit it tested (from git locally, from the report's metadata for CI
|
|
617
|
-
runs). The `commits` view uses that to say where a test went bad: the first failing or flaky
|
|
618
|
-
run after its last streak of passes, with the last good commit and a `git diff` to compare
|
|
619
|
-
them.
|
|
620
|
-
|
|
621
|
-
### `load-report`
|
|
622
|
-
|
|
623
|
-
| Argument | Type | Description |
|
|
624
|
-
| --- | --- | --- |
|
|
625
|
-
| `report` | string | A Playwright JSON report inside the project (e.g. `ci-artifacts/results.json`), or a folder of blob-report zips from sharded jobs (merged with the project's `playwright merge-reports`) (required) |
|
|
626
|
-
| `traceDirs` | string[] | Folders inside the project where the job's `test-results` artifact was unpacked (default: the report's folder and its parent) |
|
|
627
|
-
| `commit` / `branch` | string | Override what the report's metadata says (Playwright records it with `captureGitInfo` or on CI) |
|
|
628
|
-
| `record` | boolean | Add the run to the history (default `true`) |
|
|
629
|
-
| `timeoutMs` | number | Limit for merging blob reports (default `120000`) |
|
|
194
|
+
Or paste the standard config into `.gemini/settings.json` (Qwen Code: `.qwen/settings.json`).
|
|
630
195
|
|
|
631
|
-
|
|
632
|
-
`gh run download <run-id> -n playwright-report -D ci-artifacts`. The CI machine's absolute
|
|
633
|
-
paths are mapped onto your checkout, so failures point at local files and get-failure can
|
|
634
|
-
open each trace. The run becomes the last run, so the usual loop works on it: `get-failure`,
|
|
635
|
-
then `suggest-fix` (which re-runs the fix locally), or `diagnose-flaky`.
|
|
196
|
+
</details>
|
|
636
197
|
|
|
637
|
-
|
|
198
|
+
<details>
|
|
199
|
+
<summary><b>Windsurf, Cline, Zed and other clients</b></summary>
|
|
638
200
|
|
|
639
|
-
|
|
201
|
+
Add the standard config to the client's MCP config file (Cline: *MCP Servers → Configure*).
|
|
202
|
+
Clients that take a command, such as Zed, run `npx` with the arguments `-y playwright-e2e-mcp`.
|
|
640
203
|
|
|
641
|
-
|
|
204
|
+
</details>
|
|
642
205
|
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
(`npx playwright install chromium`)
|
|
206
|
+
<details>
|
|
207
|
+
<summary><b>Freebuff / Codebuff</b></summary>
|
|
646
208
|
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
|
|
209
|
+
Drop an `mcp.json` with the standard config into your project's `.agents/` folder. This repo
|
|
210
|
+
ships one in [`.agents/mcp.json`](.agents/mcp.json), so opening the checkout attaches the
|
|
211
|
+
server workspace-wide. Freebuff asks you to trust a repository's `.agents/` on first run.
|
|
650
212
|
|
|
651
|
-
|
|
652
|
-
npx -y github:xsignon-secure/E2E#v0.5.0
|
|
653
|
-
npm install -D github:xsignon-secure/E2E#v0.5.0 @playwright/test # or as a project dependency
|
|
654
|
-
npx playwright install chromium
|
|
655
|
-
```
|
|
213
|
+
</details>
|
|
656
214
|
|
|
657
|
-
|
|
658
|
-
|
|
215
|
+
<details>
|
|
216
|
+
<summary><b>ChatGPT and other remote-only clients</b></summary>
|
|
659
217
|
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
218
|
+
Some clients only accept a remote HTTPS server. This repo ships a Streamable HTTP bridge you
|
|
219
|
+
can host for free on Vercel, with OAuth sign-in for claude.ai connectors. See
|
|
220
|
+
[Hosted endpoint](docs/hosted-endpoint.md). On Vercel the page tools (inspect a page, check
|
|
221
|
+
selectors, accessibility and Web Vitals audits) run on a bundled Chromium against public URLs.
|
|
222
|
+
Running and fixing your own tests needs your project, so use the local install for that.
|
|
663
223
|
|
|
664
|
-
|
|
665
|
-
`io.github.xsignon-secure/E2E` — registry-aware clients discover it there, and every
|
|
666
|
-
`v*` release tag republishes the entry from CI via [`server.json`](server.json).
|
|
224
|
+
</details>
|
|
667
225
|
|
|
668
|
-
|
|
226
|
+
<details>
|
|
227
|
+
<summary><b>Other ways to install</b></summary>
|
|
669
228
|
|
|
670
|
-
|
|
229
|
+
| Route | How |
|
|
230
|
+
| --- | --- |
|
|
231
|
+
| Skill only, any agent | `npx skills add xsignon-secure/E2E` |
|
|
232
|
+
| MCP Registry | `io.github.xsignon-secure/E2E` ([listing](https://registry.modelcontextprotocol.io/)); registry-aware clients find it there |
|
|
233
|
+
| Straight from GitHub, no npm | `npx -y github:xsignon-secure/E2E#v0.5.1` (pin a release tag) |
|
|
234
|
+
| Release tarball | `npm install -D https://github.com/xsignon-secure/E2E/releases/download/v0.5.1/playwright-e2e-mcp-0.5.1.tgz` |
|
|
235
|
+
| Docker | `docker run -i --rm -v "$PWD:/work" playwright-e2e-mcp` (see [Development](docs/development.md#docker)) |
|
|
236
|
+
| Local checkout | `"command": "node", "args": ["/path/to/E2E/dist/index.js"]` |
|
|
671
237
|
|
|
672
|
-
|
|
673
|
-
{
|
|
674
|
-
"mcpServers": {
|
|
675
|
-
"playwright-e2e": {
|
|
676
|
-
"command": "npx",
|
|
677
|
-
"args": ["-y", "github:xsignon-secure/E2E#v0.5.0"],
|
|
678
|
-
"env": { "PW_MCP_PROJECT_ROOT": "/absolute/path/to/your/project" }
|
|
679
|
-
}
|
|
680
|
-
}
|
|
681
|
-
}
|
|
682
|
-
```
|
|
238
|
+
</details>
|
|
683
239
|
|
|
684
|
-
|
|
685
|
-
(Cline: *MCP Servers → Configure*). **Zed** and other clients that take a command: run `npx`
|
|
686
|
-
with the arguments `-y playwright-e2e-mcp`.
|
|
687
|
-
The server uses its working directory as the project root; set `PW_MCP_PROJECT_ROOT`
|
|
688
|
-
when the client launches it somewhere else (e.g. your home directory).
|
|
240
|
+
### Your first prompt
|
|
689
241
|
|
|
690
|
-
|
|
242
|
+
Restart your client, then ask:
|
|
691
243
|
|
|
692
|
-
```bash
|
|
693
|
-
codex mcp add playwright-e2e -- npx -y github:xsignon-secure/E2E#v0.5.0
|
|
694
|
-
code --add-mcp '{"name":"playwright-e2e","command":"npx","args":["-y","github:xsignon-secure/E2E#v0.5.0"]}'
|
|
695
244
|
```
|
|
696
|
-
|
|
697
|
-
Codex's defaults fight this server: the first launch clones the repo and runs
|
|
698
|
-
`tsc` (measured 30 s on a cold `npx` cache, against a 10 s
|
|
699
|
-
`startup_timeout_sec` default), and a Playwright run with retries beats the 60 s
|
|
700
|
-
`tool_timeout_sec` default. Raise both in `~/.codex/config.toml`:
|
|
701
|
-
|
|
702
|
-
```toml
|
|
703
|
-
[mcp_servers.playwright-e2e]
|
|
704
|
-
command = "npx"
|
|
705
|
-
args = ["-y", "github:xsignon-secure/E2E#v0.5.0"]
|
|
706
|
-
startup_timeout_sec = 60
|
|
707
|
-
tool_timeout_sec = 600
|
|
245
|
+
Run my Playwright tests and fix whatever fails.
|
|
708
246
|
```
|
|
709
247
|
|
|
710
|
-
|
|
711
|
-
|
|
712
|
-
the bundle ships its own dependencies, so no Node setup is required. On install it
|
|
713
|
-
asks you to pick your **project root** (required, no default: choose the project
|
|
714
|
-
folder, not your home directory) and wires it
|
|
715
|
-
into `PW_MCP_PROJECT_ROOT`, so the tools point at a real project from the first call.
|
|
248
|
+
Your agent should run the suite, read the failure, and propose a fix it has re-run. To check the
|
|
249
|
+
setup without your own tests, ask it to run `examples/sample-test.spec.ts` from this repo.
|
|
716
250
|
|
|
717
|
-
|
|
251
|
+
## What's new in 0.5.0
|
|
718
252
|
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
253
|
+
- **Diagnose CI failures:** `load-report` reads a CI job's report (or sharded blob reports)
|
|
254
|
+
and its traces, so the same diagnosis and verified fixes work on failures from CI.
|
|
255
|
+
- **Know which commit broke a test:** `analyze-history` adds `commits`, `slow` and `trend` views.
|
|
256
|
+
- **Write tests by walking the flow:** `generate-e2e-test` with `steps` runs them on the live
|
|
257
|
+
page, writes the spec with end-state assertions, and runs it once to prove it passes.
|
|
258
|
+
- **Performance audits:** `audit-performance` measures Web Vitals and hands back a budget test.
|
|
259
|
+
- **More page control:** aria snapshots with ready `toMatchAriaSnapshot` assertions, network
|
|
260
|
+
mocks, HAR record/replay, and Chromium, Firefox or WebKit.
|
|
722
261
|
|
|
723
|
-
|
|
724
|
-
`.gemini/settings.json` (Qwen Code: `.qwen/settings.json`) — both speak the same
|
|
725
|
-
MCP settings format.
|
|
262
|
+
[All release notes](https://github.com/xsignon-secure/E2E/releases)
|
|
726
263
|
|
|
727
|
-
|
|
728
|
-
[`.agents/mcp.json`](.agents/mcp.json), so opening the checkout in Freebuff
|
|
729
|
-
attaches the server workspace-wide — no global config needed. Your own projects
|
|
730
|
-
can do the same: drop an `mcp.json` with the block above into their `.agents/`
|
|
731
|
-
directory. Freebuff asks you to trust a repository's `.agents/` on first run.
|
|
732
|
-
|
|
733
|
-
All tools ship MCP **tool annotations** (`readOnlyHint`, `destructiveHint`,
|
|
734
|
-
`idempotentHint`, `openWorldHint`), so clients can show accurate safety prompts
|
|
735
|
-
before running anything.
|
|
736
|
-
|
|
737
|
-
**From a local checkout:**
|
|
264
|
+
## Tools
|
|
738
265
|
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
749
|
-
|
|
266
|
+
| Tool | Purpose |
|
|
267
|
+
| --- | --- |
|
|
268
|
+
| `run-test` | Run Playwright tests and return stats, failures, diagnostics and hints; reports progress, and `background: true` returns a run id for long suites |
|
|
269
|
+
| `get-run-status` | Progress or final result of a background run; can wait for it or cancel it |
|
|
270
|
+
| `get-failure` | Deep analysis of one failure: stack, expected/actual, **DOM snapshot at failure (from the Playwright trace)**, failed requests, console errors, next steps |
|
|
271
|
+
| `load-report` | Load a run from CI: a JSON report or a folder of sharded blob reports; CI paths are mapped onto the checkout and traces found in the unpacked artifact, so `get-failure`, `suggest-fix` and `diagnose-flaky` work on it, and the run joins the history with its commit |
|
|
272
|
+
| `suggest-fix` | Patch a failure: a drifted locator is replaced with one **proven unique** on the DOM at failure, an ambiguous one (strict mode violation) gets the unique locator of the intended match, changed copy gets the new expected text; `apply: true` writes it, re-runs the test and reverts if still red |
|
|
273
|
+
| `inspect-page` | Open a URL headlessly (optionally signed in, after steps) and return elements with **verified Playwright locators**, CSS selectors, visibility, text and console output, or the **accessibility snapshot with a ready `toMatchAriaSnapshot` assertion** |
|
|
274
|
+
| `list-tests` | List available tests (`file`, `line`, full title, projects) with filtering |
|
|
275
|
+
| `validate-selector` | Check a CSS selector, Playwright selector or **locator expression** against a live page: validity, match count, and a sturdier verified locator for each match |
|
|
276
|
+
| `generate-e2e-test` | Scaffold a Playwright test from a description using the project's **real** selectors and verified locators from the live page, or **walk a flow** step by step on the live page and write a spec that is run once to prove it passes |
|
|
277
|
+
| `compare-visual-state` | Visual regression: screenshot before/after a change and report *what* moved and how colors shifted |
|
|
278
|
+
| `audit-accessibility` | Run axe-core (WCAG 2.2 AA by default) on a page, signed in or after steps, and return violations worst first with **verified locators** for the offending elements and a regression check |
|
|
279
|
+
| `audit-performance` | Measure lab **Web Vitals** (LCP, CLS, TBT, FCP, TTFB, INP during steps), optionally as a throttled phone and the median of several runs; rate them or check budgets, name the LCP and shifting elements as verified locators, list heavy and render-blocking requests, and return a budget test |
|
|
280
|
+
| `diagnose-flaky` | Run a failing test 2–10 times **with retries disabled** and return an evidence verdict: `CONSISTENTLY FAILING`, `FLAKY` or `NOT REPRODUCING`; tells parallelism and **test-order** bugs from races |
|
|
281
|
+
| `analyze-history` | Rank flaky, regressed and broken tests from the local run history, group failures by cause, list the slowest tests, show stability by day, name **the commit where each test went bad**, or show one test's timeline |
|
|
750
282
|
|
|
751
|
-
|
|
283
|
+
Two **prompts** package the main workflows for clients that show them as slash commands:
|
|
284
|
+
`fix-failing-test` and `triage-flaky-tests`.
|
|
752
285
|
|
|
753
|
-
|
|
754
|
-
|
|
755
|
-
|
|
286
|
+
Every argument and example output is in the **[tool reference](docs/tools.md)**. All tools carry
|
|
287
|
+
MCP tool annotations (`readOnlyHint`, `destructiveHint`, …), so clients can show accurate safety
|
|
288
|
+
prompts.
|
|
756
289
|
|
|
757
|
-
|
|
758
|
-
|
|
759
|
-
| **Endpoint** | `https://playwright-e2e-mcp.vercel.app/api/mcp` |
|
|
760
|
-
| **Transport** | MCP Streamable HTTP (`POST` JSON in, JSON or SSE out) |
|
|
761
|
-
| **Auth** | bearer token (`PW_MCP_HTTP_TOKEN`), or OAuth sign-in with that token as the access key (for claude.ai connectors). The deployment at this URL requires one of them; a bridge started without the variable serves only the read-only tools |
|
|
762
|
-
| **Source** | [`api/mcp.ts`](api/mcp.ts) → [`src/http.ts`](src/http.ts) |
|
|
763
|
-
|
|
764
|
-
The bridge runs the *same* `createServer()` as the stdio transport; the SDK
|
|
765
|
-
serves every request with a fresh server instance, which is what a serverless
|
|
766
|
-
function wants. `test/http-bridge.test.mjs` drives the real Node adapter over
|
|
767
|
-
`node:http` so a broken bridge fails in CI, not in ChatGPT.
|
|
768
|
-
|
|
769
|
-
**This demo deployment is token-protected.** `https://playwright-e2e-mcp.vercel.app`
|
|
770
|
-
runs on the maintainer's account and has `PW_MCP_HTTP_TOKEN` set, so a request
|
|
771
|
-
without `Authorization: Bearer <token>` (or an OAuth sign-in with it) gets
|
|
772
|
-
`401 {"error":"unauthorized"}`. Nothing is held back: the same bridge is free to
|
|
773
|
-
host yourself on a free Vercel account, with your own token:
|
|
774
|
-
|
|
775
|
-
[](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fxsignon-secure%2FE2E&env=PW_MCP_HTTP_TOKEN&envDescription=A%20random%20secret%20of%2016%2B%20characters%3B%20clients%20send%20it%20as%20a%20bearer%20token%20or%20type%20it%20to%20sign%20in&project-name=playwright-e2e-mcp)
|
|
776
|
-
|
|
777
|
-
A serverless function has no browser, so for running and fixing your own tests
|
|
778
|
-
the free local install (`npx -y playwright-e2e-mcp init`) is the better route.
|
|
779
|
-
|
|
780
|
-
**Open vs. token-protected.** With `PW_MCP_HTTP_TOKEN` set (at least 16
|
|
781
|
-
characters; use a random value) every tool is served to clients that send
|
|
782
|
-
`Authorization: Bearer <token>`, and every other request gets `401`. Without that
|
|
783
|
-
variable anyone who reaches the URL can call the bridge, so it degrades to
|
|
784
|
-
`list-tests` and `get-failure` in restricted mode: `list-tests` scans sources
|
|
785
|
-
instead of running `playwright test --list` (which would execute the project's
|
|
786
|
-
config), callers cannot pick another `projectRoot`, and nothing spawns a process,
|
|
787
|
-
drives a browser or writes a file. Child processes started over HTTP get only an
|
|
788
|
-
allowlisted environment. `PW_MCP_ALLOWED_HOSTS` (comma separated, `*` for any)
|
|
789
|
-
limits the accepted `Host` header; without a token and without that variable,
|
|
790
|
-
only `localhost` names and the deployment's own Vercel hostnames are accepted
|
|
791
|
-
(DNS-rebinding protection).
|
|
792
|
-
|
|
793
|
-
**Add it to ChatGPT:** Settings → Connectors → turn on **Advanced → Developer
|
|
794
|
-
mode** → *Create custom connector* → paste the endpoint above. Authentication
|
|
795
|
-
**None** only works against a bridge with no token (read-only tools), so for this
|
|
796
|
-
deployment choose the connector's **API key** authentication and send
|
|
797
|
-
`Authorization: Bearer <token>` — or point the connector at your own deployment
|
|
798
|
-
with `PW_MCP_HTTP_TOKEN` unset.
|
|
799
|
-
|
|
800
|
-
**Add it to claude.ai (or Claude Desktop's connectors):** Settings → Connectors →
|
|
801
|
-
*Add custom connector* → paste the endpoint above and leave the OAuth fields
|
|
802
|
-
empty. Claude registers itself and opens a sign-in page; type the deployment's
|
|
803
|
-
access key (the value of `PW_MCP_HTTP_TOKEN`) and press **Allow access**. The
|
|
804
|
-
bridge is its own small OAuth 2.1 server (discovery, dynamic client
|
|
805
|
-
registration, authorization code with PKCE S256, refresh tokens) and keeps no
|
|
806
|
-
state: client ids, codes and tokens are signed with a key derived from
|
|
807
|
-
`PW_MCP_HTTP_TOKEN`, access tokens last an hour and refresh tokens 30 days, and
|
|
808
|
-
changing `PW_MCP_HTTP_TOKEN` signs every connected client out. The sign-in page
|
|
809
|
-
shows which site will receive access, so only type the key when that is the
|
|
810
|
-
app you are connecting. Source: [`src/oauth.ts`](src/oauth.ts); the
|
|
811
|
-
`/.well-known/oauth-*` and `/oauth/*` paths reach the function through the
|
|
812
|
-
rewrites in [`vercel.json`](vercel.json).
|
|
813
|
-
|
|
814
|
-
To self-host the open, read-only variant, deploy this repo with
|
|
815
|
-
`PW_MCP_HTTP_TOKEN` unset; nothing else changes.
|
|
816
|
-
|
|
817
|
-
Codex can also take the remote transport instead of spawning `npx`, if you'd
|
|
818
|
-
rather not ship Playwright to every machine. Point it at the token through an
|
|
819
|
-
environment variable so the secret stays out of the config file:
|
|
290
|
+
**Only need the fix loop?** `--tools=core` (or `PW_MCP_TOOLS=core`) exposes six tools and cuts the
|
|
291
|
+
tool-definition context by about three quarters. Other profiles: `triage`, `inspect`, `author`.
|
|
820
292
|
|
|
821
|
-
|
|
822
|
-
|
|
823
|
-
codex mcp add playwright-e2e-remote --url https://playwright-e2e-mcp.vercel.app/api/mcp \
|
|
824
|
-
--bearer-token-env-var PW_MCP_HTTP_TOKEN
|
|
825
|
-
```
|
|
293
|
+
<details>
|
|
294
|
+
<summary><b>Typical workflow</b></summary>
|
|
826
295
|
|
|
827
|
-
|
|
828
|
-
|
|
829
|
-
|
|
830
|
-
|
|
831
|
-
|
|
832
|
-
|
|
833
|
-
|
|
296
|
+
1. `run-test` `{ "testFiles": ["tests/checkout.spec.ts"] }`, or `{ "background": true }`
|
|
297
|
+
for a long suite and `get-run-status` `{ "waitSeconds": 30 }` until it finishes. For a
|
|
298
|
+
failure from CI, download the job's artifacts and call `load-report`
|
|
299
|
+
`{ "report": "ci-artifacts/results.json" }` instead.
|
|
300
|
+
2. `get-failure` `{ "index": 1 }` to read the code frame, expected/actual, **the DOM
|
|
301
|
+
snapshot at failure**, the failed network requests and the page's console errors.
|
|
302
|
+
3. `suggest-fix` `{ "index": 1 }` for a locator or copy change: review the diff, then
|
|
303
|
+
`{ "index": 1, "apply": true }` to write it and verify with a re-run. If it reports the
|
|
304
|
+
element is missing, fix the app, not the test.
|
|
305
|
+
4. For anything it cannot patch: `inspect-page` `{ "url": "…", "view": "locators" }`
|
|
306
|
+
(add `storageState` or `actions` to reach the right screen), then `validate-selector`
|
|
307
|
+
to prove the new locator before editing.
|
|
308
|
+
5. If a failure looks intermittent: `diagnose-flaky` `{ "runs": 3 }` for fresh evidence,
|
|
309
|
+
and `analyze-history` to see how the test behaved across earlier runs
|
|
310
|
+
(`{ "view": "commits" }` names the commit where it went bad).
|
|
311
|
+
6. After changing CSS or components: `compare-visual-state` `{ "url": "…", "name": "checkout" }`.
|
|
312
|
+
7. Re-run **only what failed** with `run-test` `{ "lastFailed": true }` until green.
|
|
834
313
|
|
|
835
|
-
|
|
836
|
-
|
|
837
|
-
|
|
838
|
-
|
|
839
|
-
-H 'content-type: application/json' \
|
|
840
|
-
-H 'accept: application/json, text/event-stream' \
|
|
841
|
-
-H "authorization: Bearer $PW_MCP_HTTP_TOKEN" \
|
|
842
|
-
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}}'
|
|
843
|
-
```
|
|
314
|
+
To write a new test, find the locators with `inspect-page` `{ "view": "locators" }`, then call
|
|
315
|
+
`generate-e2e-test` with a `description` and the `steps` of the flow; it walks them on the live
|
|
316
|
+
page, writes the spec and runs it once. To check a page's speed or accessibility, use
|
|
317
|
+
`audit-performance` and `audit-accessibility`.
|
|
844
318
|
|
|
845
|
-
|
|
846
|
-
every push, so there is no token to manage and no CLI step — watch the
|
|
847
|
-
`Vercel` commit status for the deployment result.
|
|
319
|
+
</details>
|
|
848
320
|
|
|
849
321
|
## CI: GitHub Action and `ci` command
|
|
850
322
|
|
|
@@ -872,7 +344,7 @@ steps:
|
|
|
872
344
|
- uses: actions/setup-node@v6
|
|
873
345
|
with: { node-version: 24 }
|
|
874
346
|
- run: npm ci && npx playwright install --with-deps chromium
|
|
875
|
-
- uses: xsignon-secure/E2E@v0.5.
|
|
347
|
+
- uses: xsignon-secure/E2E@v0.5.1
|
|
876
348
|
with:
|
|
877
349
|
flaky-runs: 3 # 0 skips the flaky verdict
|
|
878
350
|
suggest: 2 # suggest-fix diffs for the first 2 failures
|
|
@@ -882,20 +354,11 @@ steps:
|
|
|
882
354
|
Inputs: `project-root`, `args`, `flaky-runs`, `suggest`, `comment`,
|
|
883
355
|
`fail-on-failure`, `version`, `github-token`. Outputs: `exit-code`, `report`.
|
|
884
356
|
|
|
885
|
-
## Docker
|
|
886
|
-
|
|
887
|
-
```bash
|
|
888
|
-
docker build -t playwright-e2e-mcp .
|
|
889
|
-
docker run -i --rm -v "$PWD:/work" playwright-e2e-mcp # stdio MCP server
|
|
890
|
-
docker run --rm -v "$PWD:/work" playwright-e2e-mcp ci # CI report
|
|
891
|
-
```
|
|
892
|
-
|
|
893
|
-
The image is based on `mcr.microsoft.com/playwright:v1.63.0-noble`, so its
|
|
894
|
-
browsers match Playwright 1.63. Mount your project at `/work`; its own
|
|
895
|
-
`node_modules` provide `@playwright/test`.
|
|
896
|
-
|
|
897
357
|
## Configuration
|
|
898
358
|
|
|
359
|
+
<details>
|
|
360
|
+
<summary><b>Environment variables</b></summary>
|
|
361
|
+
|
|
899
362
|
| Environment variable | Default | Purpose |
|
|
900
363
|
| --- | --- | --- |
|
|
901
364
|
| `PW_MCP_PROJECT_ROOT` | server cwd | Default project root for every tool |
|
|
@@ -904,11 +367,12 @@ browsers match Playwright 1.63. Mount your project at `/work`; its own
|
|
|
904
367
|
| `PW_MCP_SHUFFLE_SEEDS` | random | Comma-separated seeds for `diagnose-flaky`'s `order` pass, to replay the orders a diagnosis reported |
|
|
905
368
|
| `PW_MCP_HISTORY` | on | `0` stops recording runs to `.playwright-e2e-mcp/history.jsonl` |
|
|
906
369
|
| `PW_MCP_ALLOWED_ROOTS` | — | Extra directories a caller may pass as `projectRoot` (`:`-separated, `;` on Windows). Anything outside these and the default root is rejected |
|
|
907
|
-
| `PW_MCP_HTTP_TOKEN` | — | HTTP bridge only: bearer token (16+ characters) that unlocks all tools (see
|
|
370
|
+
| `PW_MCP_HTTP_TOKEN` | — | HTTP bridge only: bearer token (16+ characters) that unlocks all tools (see [Hosted endpoint](docs/hosted-endpoint.md)) |
|
|
908
371
|
| `PW_MCP_ALLOWED_HOSTS` | localhost + Vercel hostnames when there is no token | HTTP bridge only: comma-separated `Host` allowlist; `*` accepts any |
|
|
909
372
|
| `PW_MCP_PASSTHROUGH_ENV` | — | HTTP bridge only: comma-separated extra variables passed to test runs (e.g. `BASE_URL`) |
|
|
910
373
|
| `PW_MCP_MAX_CHILDREN` | `4` | HTTP bridge only: how many test runs and browser probes may run at once |
|
|
911
374
|
| `PW_MCP_BLOCK_PRIVATE_URLS` | off (on for the HTTP bridge) | `1` makes the URL tools refuse loopback, private-network and cloud-metadata addresses, including redirects and subresources |
|
|
375
|
+
| `PW_MCP_SERVERLESS_CHROMIUM` | on for Vercel | Launch the page tools on the bundled serverless Chromium (`@sparticuz/chromium`, Linux x64, Node 22+); `0` turns it off, `1` turns it on elsewhere |
|
|
912
376
|
| `LOG_LEVEL` / `MCP_LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` \| `silent`; `LOG_LEVEL` wins when both are set |
|
|
913
377
|
| `LOG_FORMAT` / `MCP_LOG_FORMAT` | `text` | `text`, or `json` (also `ndjson`) for one structured JSON object per line — ready for a log ingester |
|
|
914
378
|
|
|
@@ -921,24 +385,7 @@ object (`time`, `level`, `message` plus context):
|
|
|
921
385
|
{"time":"2026-10-07T08:00:00.000Z","level":"info","message":"server ready","name":"playwright-e2e-mcp"}
|
|
922
386
|
```
|
|
923
387
|
|
|
924
|
-
|
|
925
|
-
|
|
926
|
-
1. `run-test` `{ "testFiles": ["tests/checkout.spec.ts"] }`, or `{ "background": true }`
|
|
927
|
-
for a long suite and `get-run-status` `{ "waitSeconds": 30 }` until it finishes.
|
|
928
|
-
2. `get-failure` `{ "index": 1 }` to read the code frame, expected/actual, **the DOM
|
|
929
|
-
snapshot at failure**, the failed network requests and the page's console errors.
|
|
930
|
-
3. `suggest-fix` `{ "index": 1 }` for a locator or copy change: review the diff, then
|
|
931
|
-
`{ "index": 1, "apply": true }` to write it and verify with a re-run. If it reports the
|
|
932
|
-
element is missing, fix the app, not the test.
|
|
933
|
-
4. For anything it cannot patch: `inspect-page` `{ "url": "…", "view": "locators" }`
|
|
934
|
-
(add `storageState` or `actions` to reach the right screen), then `validate-selector`
|
|
935
|
-
to prove the new locator before editing.
|
|
936
|
-
5. If a failure looks intermittent: `diagnose-flaky` `{ "runs": 3 }` for fresh evidence,
|
|
937
|
-
and `analyze-history` to see how the test behaved across earlier runs.
|
|
938
|
-
6. After changing CSS or components: `compare-visual-state` `{ "url": "…", "name": "checkout" }`.
|
|
939
|
-
7. Re-run **only what failed** with `run-test` `{ "lastFailed": true }` until green.
|
|
940
|
-
|
|
941
|
-
To write a new test, start with `generate-e2e-test` `{ "description": "checkout with a saved card" }`.
|
|
388
|
+
</details>
|
|
942
389
|
|
|
943
390
|
## Benchmark
|
|
944
391
|
|
|
@@ -958,309 +405,34 @@ apply) and with Playwright MCP plus a shell (`playwright test`, `browser_navigat
|
|
|
958
405
|
Tokens are characters ÷ 4. The Playwright MCP route is a lower bound: it assumes the agent
|
|
959
406
|
picks the right locator from the snapshot on the first try, which the server does for it here.
|
|
960
407
|
|
|
961
|
-
##
|
|
962
|
-
|
|
963
|
-
| Situation | Behaviour |
|
|
964
|
-
| --- | --- |
|
|
965
|
-
| No Playwright installed | `NO_PLAYWRIGHT` error with the exact install commands for your package manager |
|
|
966
|
-
| Dev server not running | Failure classified `server-unreachable` / `SERVER_NOT_RUNNING` with a "start your dev server" hint (and `webServer` advice) |
|
|
967
|
-
| Test exceeds `timeoutMs` | Process **group** is killed (SIGINT→SIGKILL on POSIX, `taskkill /T /F` on Windows) and partial results are returned |
|
|
968
|
-
| Browser crashes | Classified `browser-crash` with retry / reinstall guidance |
|
|
969
|
-
| Windows backslashes | All paths normalized lexically (`C:\a\..\b` → `C:/b`); unit-tested on both platforms |
|
|
970
|
-
| Flaky tests | Failing tests are automatically retried once (`--retries=1`) before being reported; passes surface as **flaky** with a stability warning; `diagnose-flaky` decides flaky-vs-broken with multi-run evidence |
|
|
971
|
-
| Trace/DOM context | `--trace=retain-on-failure` is passed automatically, so `get-failure` can show the exact DOM at the moment of failure — plus the failed network requests (`*.network` logs) and console errors from the same trace |
|
|
972
|
-
| Slow re-runs after a fix | `run-test` with `lastFailed: true` re-runs only the tests that failed last time (`--last-failed`) |
|
|
973
|
-
| Several `playwright.config` files | Returns a numbered menu (`MULTIPLE_CONFIGS`); pick with `config: "2"` or a path |
|
|
974
|
-
| Syntax error in a spec | `SYNTAX_ERROR` with file:line; nothing crashes; `list-tests` falls back to a source scan |
|
|
975
|
-
| MCP client disconnects | Per-request `AbortSignal` kills the run; stdin end triggers shutdown, and every tracked child tree is force-killed (`killActiveChildren`) |
|
|
976
|
-
| Disk full | `ENOSPC` detected → `DISK_FULL` with a "free space" hint; logging never throws |
|
|
977
|
-
| Malicious paths | `../../etc/passwd`, absolute paths outside the root, a `projectRoot` outside the allowed roots, symlinks that leave the root, URLs and null bytes are rejected with `INVALID_PATH`; extra CLI args must be allowlisted Playwright flags |
|
|
978
|
-
|
|
979
|
-
## Security notes
|
|
980
|
-
|
|
981
|
-
- **No shell**: Playwright is spawned as `node <playwright/cli.js> …` with an argument
|
|
982
|
-
array — no command interpolation. Extra `args` are limited to an allowlist of
|
|
983
|
-
Playwright flags, each with a checked value (`--flag=value`), so a caller cannot
|
|
984
|
-
swap in another `--config` or `--output` or hand git an option through
|
|
985
|
-
`--only-changed`. Paths with a segment starting with `-` are rejected, so a test
|
|
986
|
-
file name cannot be read as a flag.
|
|
987
|
-
- **Path sandbox**: user paths must stay inside the project root, checked lexically
|
|
988
|
-
and again with symlinks resolved. A caller-supplied `projectRoot` must sit inside
|
|
989
|
-
`PW_MCP_PROJECT_ROOT` (or a `PW_MCP_ALLOWED_ROOTS` entry).
|
|
990
|
-
- **Generated code**: `generate-e2e-test` only writes `*.spec.*` / `*.test.*` files,
|
|
991
|
-
only overwrites specs whose header it wrote itself, never writes through a symlink,
|
|
992
|
-
and escapes every value it puts into strings or comments.
|
|
993
|
-
- **HTTP bridge**: read-only tools unless `PW_MCP_HTTP_TOKEN` is set; child processes
|
|
994
|
-
started over HTTP get only an allowlisted environment (`PATH`, `HOME`, temp dirs,
|
|
995
|
-
locale, `CI`, `PLAYWRIGHT_*`, `npm_config_*` without embedded credentials, plus
|
|
996
|
-
anything in `PW_MCP_PASSTHROUGH_ENV`), at most `PW_MCP_MAX_CHILDREN` run at once
|
|
997
|
-
(a disconnected client frees its slot immediately), and internal error messages
|
|
998
|
-
are not returned to clients. The allowlist only covers the child's own
|
|
999
|
-
environment: test code runs as the same OS user, so on Linux it could still read
|
|
1000
|
-
the server's startup environment from `/proc`. That is why only token holders
|
|
1001
|
-
can run project code; keep other secrets out of the bridge's environment, or run
|
|
1002
|
-
it under a separate user.
|
|
1003
|
-
- **SSRF guard**: on the HTTP bridge (or with `PW_MCP_BLOCK_PRIVATE_URLS=1`) the URL tools
|
|
1004
|
-
refuse hosts that resolve to loopback, private, link-local/metadata or reserved
|
|
1005
|
-
addresses, and route the browser through a local proxy that applies the same check
|
|
1006
|
-
to every redirect hop and subresource; IPv6 forms that embed an IPv4 address
|
|
1007
|
-
(mapped, NAT64, 6to4, Teredo) are blocked too, and WebRTC UDP is disabled so a
|
|
1008
|
-
page cannot reach the network around the proxy. It is off for stdio by default because
|
|
1009
|
-
opening `http://localhost` dev servers is what these tools are for.
|
|
1010
|
-
- **Cleanup**: temp report/script files are written to the OS temp dir and removed;
|
|
1011
|
-
child processes are tracked and killed on shutdown. Trace decompression has one
|
|
1012
|
-
size budget per archive, and PNGs are capped at 16,384 px per side and 50 M pixels.
|
|
1013
|
-
|
|
1014
|
-
## Development
|
|
1015
|
-
|
|
1016
|
-
```
|
|
1017
|
-
src/
|
|
1018
|
-
├── index.ts # bin entry point (--version/--help, main-module guard)
|
|
1019
|
-
├── server.ts # McpServer setup, tool registration, shutdown handling
|
|
1020
|
-
├── tools/ # the fourteen tools, the probe script + shared plumbing
|
|
1021
|
-
├── prompts.ts # fix-failing-test, triage-flaky-tests
|
|
1022
|
-
├── utils/ # playwright-runner, report-parser, project-detector, path-utils,
|
|
1023
|
-
│ # logger, trace-reader (trace.zip → DOM/network/console),
|
|
1024
|
-
│ # image-diff (PNG codec + pixel diff), change-analyzer,
|
|
1025
|
-
│ # locator-expr, run-history, run-registry
|
|
1026
|
-
└── types/ # shared interfaces and the ErrorKind taxonomy
|
|
1027
|
-
```
|
|
1028
|
-
|
|
1029
|
-
Built on **`@modelcontextprotocol/server` v2** (the `2026-07-28` MCP spec line) with
|
|
1030
|
-
Zod v4 standard schemas; every tool declares spec tool annotations.
|
|
1031
|
-
|
|
1032
|
-
```bash
|
|
1033
|
-
npm install
|
|
1034
|
-
npm run build # tsc → dist/ (zero errors)
|
|
1035
|
-
npm test # build + test/run-tests.mjs (unit tests, any Node ≥20)
|
|
1036
|
-
npm run e2e # build + e2e/run.mjs: live MCP ↔ Playwright integration suite
|
|
1037
|
-
npm run mcpb # build + bundle dist/ into playwright-e2e-mcp-<version>.mcpb (+ .sha256)
|
|
1038
|
-
npm run mcpb:smoke # ...then install that bundle in a temp dir, launch it and assert the handshake
|
|
1039
|
-
npm run mcpb:sign # ...then sign it with MCPB_CERT/MCPB_KEY → <version>.signed.mcpb
|
|
1040
|
-
npm run skill:zip # just the archive: dist-skill/playwright-e2e-mcp.zip
|
|
1041
|
-
npm run skill:package # the uploadable folder: ../agensi-skill-<version>/ (skill copy, README, zip)
|
|
1042
|
-
npm run skill:check # build, then fail if a packaged copy has drifted from skill/
|
|
1043
|
-
```
|
|
1044
|
-
|
|
1045
|
-
Tests cover the report parser (sample Playwright JSON, trace attachments), path utils
|
|
1046
|
-
(Windows and macOS paths, sandboxing), the project detector (temp-dir fixtures: config
|
|
1047
|
-
discovery, multiple configs, missing install, test-file scanning), the shared tool
|
|
1048
|
-
helpers, the **trace reader** (synthetic trace.zip: error, failed action, DOM snapshot,
|
|
1049
|
-
`*.network` failed-request parsing, console error/warning events), the **image diff**
|
|
1050
|
-
(PNG round-trip, regions, color shift, dimension changes), the **change analyzer**
|
|
1051
|
-
(selector extraction, git + mtime paths), the **flaky verdict logic**
|
|
1052
|
-
(failure signatures, CONSISTENTLY FAILING / FLAKY / NOT REPRODUCING / NO TESTS RAN,
|
|
1053
|
-
per-test grouping of repeat attempts and the parallel-vs-alone reading), the `ci`
|
|
1054
|
-
argument parser and tool profiles,
|
|
1055
|
-
the **HTTP bridge** (token, restricted mode, Host allowlist), the **security
|
|
1056
|
-
regressions** (sandbox escapes, argument smuggling, symlink writes, code injection,
|
|
1057
|
-
SSRF ranges, env scrubbing, decode limits), the `.mcpb` manifest, the `.mcpb` zip
|
|
1058
|
-
extractor, the **skill package** (byte-identical rebuilds, a CRLF checkout hashing the
|
|
1059
|
-
same, refusal of junk/binary/symlinks, and drift detection between `skill/` and a
|
|
1060
|
-
packaged copy), and the MCP config files.
|
|
1061
|
-
|
|
1062
|
-
### Integration suite (`npm run e2e`)
|
|
1063
|
-
|
|
1064
|
-
Unit tests prove the logic; the integration suite proves the loop. It boots the real
|
|
1065
|
-
server over stdio against a live fixture app and a Playwright project under
|
|
1066
|
-
`e2e/fixture/`, then drives it exactly like an MCP client and asserts ~40 behaviours
|
|
1067
|
-
that only appear end-to-end:
|
|
1068
|
-
|
|
1069
|
-
- initialize handshake, 14 tools, 2 prompts, spec tool annotations and object input schemas,
|
|
1070
|
-
- verified locators, locator validation, signed-in pages (`storageState`) and page steps,
|
|
1071
|
-
- background runs with progress, run history views, and `suggest-fix` healing a drifted
|
|
1072
|
-
locator and an ambiguous one, verifying each with a re-run (and refusing when the
|
|
1073
|
-
element is missing),
|
|
1074
|
-
- Web Vitals with budgets and phone throttling, and network mocks that force error and
|
|
1075
|
-
empty states,
|
|
1076
|
-
- live DOM inspection, CSS selector validation (matches, zero matches, engine
|
|
1077
|
-
syntax, parse errors), dead-server detection,
|
|
1078
|
-
- visual regression: baseline → unchanged compare → `blue → red` diff detection,
|
|
1079
|
-
- `run-test` pass/fail/`lastFailed` stats and meta lines,
|
|
1080
|
-
- `get-failure` trace diagnostics: DOM at failure, the 404 network request,
|
|
1081
|
-
the `console.error` message, diagnosis and next steps,
|
|
1082
|
-
- auto-retry turning a first-run failure into `PASSED (1 flaky)`,
|
|
1083
|
-
- `diagnose-flaky` verdict **FLAKY** (2 of 3 runs) with retries disabled, in one
|
|
1084
|
-
`--repeat-each` process, and `isolate` reading a shared-slot test as "fails only in parallel",
|
|
1085
|
-
- `--tools=` profiles trimming `tools/list`, and the `ci` command's exit code, report,
|
|
1086
|
-
`--output` file and GitHub job summary,
|
|
1087
|
-
- `generate-e2e-test` writing its scaffold, plus error paths
|
|
1088
|
-
(missing test path, unknown tool, unreachable server).
|
|
1089
|
-
|
|
1090
|
-
First run needs the browser once: `npx playwright install chromium`.
|
|
1091
|
-
CI runs the suite on Ubuntu and Windows (see `.github/workflows/ci.yml`).
|
|
1092
|
-
|
|
1093
|
-
### Signing the bundle
|
|
1094
|
-
|
|
1095
|
-
```bash
|
|
1096
|
-
MCPB_CERT=~/signing/cert.pem MCPB_KEY=~/signing/key.pem npm run mcpb:sign
|
|
1097
|
-
```
|
|
1098
|
-
|
|
1099
|
-
Signing appends a PKCS#7 signature block to the bundle, so the signed file is
|
|
1100
|
-
*the reproducible bundle plus that block*. That ordering is the point: the bytes
|
|
1101
|
-
CI proved identical across platforms stay inside the signed file, and
|
|
1102
|
-
`mcpb unsign` returns them exactly. `scripts/sign-mcpb.mjs` asserts that round
|
|
1103
|
-
trip on every run and deletes its output if it fails, which is what keeps
|
|
1104
|
-
`<version>.signed.mcpb` tied to `<version>.mcpb`.
|
|
1105
|
-
|
|
1106
|
-
The private key is read from a path you pass in (`--cert`/`--key` or
|
|
1107
|
-
`MCPB_CERT`/`MCPB_KEY`) and never from the repository — keep it outside version
|
|
1108
|
-
control, and back it up: the same key across releases is what makes the signer
|
|
1109
|
-
recognisable.
|
|
1110
|
-
|
|
1111
|
-
What signing does and does not buy you is decided by the certificate, not by
|
|
1112
|
-
this script. `mcpb verify` and `mcpb info` accept a signature only when the
|
|
1113
|
-
certificate chains to something in the OS trust store, so with a self-signed
|
|
1114
|
-
certificate a host still reports **Not signed** — the signature is real
|
|
1115
|
-
integrity and key continuity, verifiable by the fingerprint the script prints,
|
|
1116
|
-
but not third-party identity. A certificate issued by a public CA, or an
|
|
1117
|
-
internal one installed on your machines, is what makes hosts report it as
|
|
1118
|
-
signed.
|
|
1119
|
-
|
|
1120
|
-
### Assembling a release
|
|
1121
|
-
|
|
1122
|
-
```bash
|
|
1123
|
-
npm run mcpb && npm pack # the bundle, then the npm tarball
|
|
1124
|
-
npm run release:assemble -- --tag v0.2.3 # sign, checksum and write the notes
|
|
1125
|
-
npm run release:publish -- --tag v0.2.3 --assets dist-release/assets \
|
|
1126
|
-
--title-file dist-release/RELEASE_TITLE.txt \
|
|
1127
|
-
--body-file dist-release/RELEASE_BODY.md --download dist-release/downloaded
|
|
1128
|
-
npm run release:verify -- v0.2.3 --dir dist-release/downloaded
|
|
1129
|
-
npm run release:publish -- --tag v0.2.3 --publish
|
|
1130
|
-
```
|
|
1131
|
-
|
|
1132
|
-
Pushing a `v*` tag runs exactly that sequence in
|
|
1133
|
-
`.github/workflows/release.yml`. The release used to be assembled by hand, and
|
|
1134
|
-
the notes were written from what those commands printed: every hash in them was
|
|
1135
|
-
a copy, and nothing compared the bundle hash printed there with the bundle that
|
|
1136
|
-
shipped.
|
|
1137
|
-
|
|
1138
|
-
`scripts/assemble-release.mjs` stages the whole release into
|
|
1139
|
-
`dist-release/assets/`: the bundle and its sidecar, the **signed** bundle and the
|
|
1140
|
-
signing certificate when a key is configured, the npm tarball, the skill package
|
|
1141
|
-
built by the same writer as `npm run skill:zip`, and `SHA256SUMS.txt` covering
|
|
1142
|
-
every file it staged. It empties that directory first, so a file left over from
|
|
1143
|
-
an earlier run cannot be hashed and published as if this run had produced it.
|
|
1144
|
-
|
|
1145
|
-
The notes are half written, half generated. `release-notes/<tag>.md` holds the
|
|
1146
|
-
prose — its H1 becomes the release title and has to name the tag — and the script
|
|
1147
|
-
appends the asset table, the reproduction recipe carrying the real bundle hash,
|
|
1148
|
-
the signer and fingerprint, and the commit list. The prose is held to the same
|
|
1149
|
-
rule: a 64-character hex string in it has to be a hash this release publishes,
|
|
1150
|
-
and writing an `### Assets` section of your own fails the assembly rather than
|
|
1151
|
-
shipping a second, staler table. See [release-notes/README.md](release-notes/README.md).
|
|
1152
|
-
|
|
1153
|
-
`scripts/publish-release.mjs` fills the release **as a draft**, uploads the
|
|
1154
|
-
assets, downloads every one of them back and compares it byte for byte, and only
|
|
1155
|
-
`--publish` makes it public — after `release:verify` has re-derived the release
|
|
1156
|
-
from the tag. Re-running is safe: a draft this workflow assembled is refreshed in
|
|
1157
|
-
place, while a release it did not assemble, or one already published, is refused
|
|
1158
|
-
by name instead of overwritten.
|
|
1159
|
-
|
|
1160
|
-
Signing is optional and its absence is stated rather than hidden: without the
|
|
1161
|
-
`MCPB_KEY` and `MCPB_CERT` repository secrets the release is assembled and
|
|
1162
|
-
published **unsigned**, with a notice in the run and a line saying so in the
|
|
1163
|
-
notes. Add both — the private key stays out of the repository and out of the logs:
|
|
1164
|
-
|
|
1165
|
-
```bash
|
|
1166
|
-
gh secret set MCPB_KEY --repo xsignon-secure/E2E < ~/signing/key.pem
|
|
1167
|
-
gh secret set MCPB_CERT --repo xsignon-secure/E2E < ~/signing/cert.pem
|
|
1168
|
-
```
|
|
1169
|
-
|
|
1170
|
-
Creating the release and uploading its assets needs `contents: write`, which the
|
|
1171
|
-
job requests for itself; if it is refused, the run names the setting to change.
|
|
408
|
+
## Demo
|
|
1172
409
|
|
|
1173
|
-
|
|
410
|
+
A real `run-test` call served over stdio by `npx -y playwright-e2e-mcp` against the bundled
|
|
411
|
+
`examples/sample-test.spec.ts` (actual output, unedited):
|
|
1174
412
|
|
|
1175
|
-
|
|
1176
|
-
git checkout v0.5.0 && npm ci && npm run mcpb
|
|
1177
|
-
npm run release:verify -- v0.5.0
|
|
1178
|
-
```
|
|
413
|
+
|
|
1179
414
|
|
|
1180
|
-
|
|
1181
|
-
byte. `scripts/verify-release.mjs` checks that promise from a checkout of the
|
|
1182
|
-
tag: the published bundle against your fresh build, every checksum the release
|
|
1183
|
-
publishes (and that nothing shippable is missing from `SHA256SUMS.txt`), a
|
|
1184
|
-
**signed** bundle against the bytes its signature wraps, the **skill package**
|
|
1185
|
-
against a rebuild from the same tag, and that the published archive still
|
|
1186
|
-
extracts with that version's manifest. The skill package is required of a tag
|
|
1187
|
-
whose tree carries the publishing workflow; older tags predate it, so there it
|
|
1188
|
-
is checked when present and skipped when not, like the signing certificate.
|
|
1189
|
-
`.github/workflows/verify-release.yml` runs it on demand for any past tag, and by
|
|
1190
|
-
itself for a release published by hand. A release assembled by
|
|
1191
|
-
`.github/workflows/release.yml` runs the same check on its own assets *before*
|
|
1192
|
-
it is published — GitHub does not raise `release: published` for a release created
|
|
1193
|
-
with the repository's own token, so the check is a step there rather than a
|
|
1194
|
-
trigger. The `.tgz` is checked for download integrity only: npm
|
|
1195
|
-
records the packer's line endings, so a tarball packed locally on another OS is
|
|
1196
|
-
legitimately different.
|
|
1197
|
-
|
|
1198
|
-
### Packaging the agent skill
|
|
415
|
+
## Free and open source
|
|
1199
416
|
|
|
1200
|
-
|
|
1201
|
-
|
|
1202
|
-
|
|
1203
|
-
|
|
1204
|
-
|
|
1205
|
-
```
|
|
417
|
+
- **MIT licensed.** Use it, change it, ship it in commercial work; no fees, no paid tier, no feature held back.
|
|
418
|
+
- **No account, no API key, no sign-up.** `npx -y playwright-e2e-mcp init` is the whole setup.
|
|
419
|
+
- **No telemetry.** It never phones home; your tests, traces and run history stay on your machine.
|
|
420
|
+
- **Free dependencies only:** the MCP SDK (Apache-2.0), zod (MIT), axe-core (MPL-2.0) and the Playwright your project already has (Apache-2.0).
|
|
421
|
+
- **Free to host yourself**, too: the HTTP bridge deploys to a free Vercel account (see [Hosted endpoint](docs/hosted-endpoint.md)).
|
|
1206
422
|
|
|
1207
|
-
|
|
1208
|
-
|
|
1209
|
-
|
|
1210
|
-
`scripts/package-skill.mjs` produces that zip, so the uploaded copy is built
|
|
1211
|
-
rather than assembled by hand, and it refuses what the marketplace rejects: a
|
|
1212
|
-
missing `SKILL.md`, a non-text file, a symlink, `Thumbs.db`.
|
|
1213
|
-
|
|
1214
|
-
`skill:package` writes the whole folder a release would otherwise assemble by
|
|
1215
|
-
hand — the skill copy, `skill/README.md` and the archive — to
|
|
1216
|
-
`../agensi-skill-<version>` (or `--out <dir>`), then verifies it with the same
|
|
1217
|
-
comparison `check` uses, so a successful exit means the folder really is the
|
|
1218
|
-
skill tree. It is idempotent: a folder that already matches is left untouched.
|
|
1219
|
-
A folder that has drifted is refused instead of overwritten, because it may
|
|
1220
|
-
already be published — bump `version` for a new package, or pass `--force` to
|
|
1221
|
-
replace that one.
|
|
1222
|
-
|
|
1223
|
-
The archive comes from the same deterministic writer as the `.mcpb`, so the
|
|
1224
|
-
same tree always hashes the same, and text payloads are LF-normalised first:
|
|
1225
|
-
a CRLF checkout builds the bytes an LF one does. CI runs `skill:check`, generates
|
|
1226
|
-
a package folder, and uploads the built zip as an artifact
|
|
1227
|
-
(`.github/workflows/ci.yml`).
|
|
1228
|
-
|
|
1229
|
-
**Every release publishes the package too.** Assembling a release builds it from
|
|
1230
|
-
the release tag and attaches it as `<name>-<version>-skill.zip`, and the
|
|
1231
|
-
`.github/workflows/release.yml` run downloads that asset back off the draft and
|
|
1232
|
-
byte-compares it with the build before anything is published. The marketplace
|
|
1233
|
-
upload is that asset, not a folder someone assembled — a deterministic archive
|
|
1234
|
-
means "the same build as the tagged release" is an exact statement, and
|
|
1235
|
-
`release:verify` checks it again against a rebuild from the tag.
|
|
1236
|
-
`.github/workflows/publish-skill-package.yml` stays for the repair path: run it
|
|
1237
|
-
by hand to attach or rebuild the package for a release that is missing it.
|
|
1238
|
-
|
|
1239
|
-
`skill:check` is the other half of that, and the part that matters when a
|
|
1240
|
-
package has already been uploaded. It builds twice (the bytes must agree),
|
|
1241
|
-
reads the archive back and compares its contents with `skill/` file by file,
|
|
1242
|
-
then compares a packaged copy — `../agensi-skill-<version>` by default, or a
|
|
1243
|
-
path you pass: the package folder, the bare `playwright-e2e-mcp/` folder, or
|
|
1244
|
-
the zip on its own. Comparisons are by content, never by archive bytes, because
|
|
1245
|
-
a zip written by a platform `zip` tool carries real timestamps that say nothing
|
|
1246
|
-
about whether the Markdown inside it is current. Any file that differs, is
|
|
1247
|
-
missing or is extra is named on stderr and the command exits 1 — the repository
|
|
1248
|
-
copy and the uploaded copy are the same document, so an edit to either one
|
|
1249
|
-
should not be invisible.
|
|
1250
|
-
|
|
1251
|
-
### Try the example
|
|
1252
|
-
|
|
1253
|
-
With Playwright installed in your project:
|
|
423
|
+
**What it does not do:** it doesn't drive an exploratory browsing session step by step, and it
|
|
424
|
+
doesn't replace Playwright's own Test Agents inside VS Code. Use Playwright MCP for interactive
|
|
425
|
+
browsing and this server for running, diagnosing and fixing the suite. They work side by side.
|
|
1254
426
|
|
|
1255
|
-
|
|
1256
|
-
npx playwright test examples/sample-test.spec.ts
|
|
1257
|
-
```
|
|
427
|
+
## More documentation
|
|
1258
428
|
|
|
1259
|
-
|
|
1260
|
-
|
|
1261
|
-
[
|
|
1262
|
-
|
|
429
|
+
- [Tool reference](docs/tools.md): every argument, example outputs, edge cases handled
|
|
430
|
+
- [Hosted endpoint](docs/hosted-endpoint.md): the HTTP bridge for ChatGPT and claude.ai connectors
|
|
431
|
+
- [Security](SECURITY.md): the security model and how to report a vulnerability
|
|
432
|
+
- [Development](docs/development.md): building, tests, Docker, signing and releasing
|
|
433
|
+
- [Contributing](CONTRIBUTING.md) · [Code of conduct](CODE_OF_CONDUCT.md)
|
|
1263
434
|
|
|
1264
435
|
## License
|
|
1265
436
|
|
|
1266
|
-
MIT
|
|
437
|
+
[MIT](LICENSE). If this saves you a debugging session, a ⭐ on the repo helps other Playwright
|
|
438
|
+
users find it.
|