@matware/e2e-runner 1.3.1 โ 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +4 -4
- package/.claude-plugin/plugin.json +2 -2
- package/LICENSE +1 -1
- package/README.md +491 -225
- package/agents/test-creator.md +4 -2
- package/agents/test-improver.md +7 -4
- package/bin/cli.js +93 -19
- package/package.json +4 -3
- package/skills/e2e-testing/SKILL.md +5 -3
- package/skills/e2e-testing/references/action-types.md +35 -18
- package/skills/e2e-testing/references/test-json-format.md +23 -0
- package/skills/e2e-testing/references/troubleshooting.md +2 -26
- package/src/actions.js +181 -15
- package/src/config.js +6 -0
- package/src/dashboard.js +185 -9
- package/src/db.js +26 -0
- package/src/mcp-tools.js +238 -69
- package/src/module-analysis.js +247 -0
- package/src/module-resolver.js +35 -2
- package/src/narrate.js +33 -1
- package/src/pool-manager.js +46 -1
- package/src/pool.js +177 -20
- package/src/runner.js +144 -19
- package/src/visual-diff.js +74 -4
- package/src/websocket.js +14 -3
- package/src/wizard.js +184 -0
- package/templates/build-dashboard.js +3 -0
- package/templates/dashboard/js/api.js +60 -3
- package/templates/dashboard/js/init.js +46 -0
- package/templates/dashboard/js/keyboard.js +8 -7
- package/templates/dashboard/js/quicksearch.js +277 -0
- package/templates/dashboard/js/state.js +61 -7
- package/templates/dashboard/js/toast.js +1 -1
- package/templates/dashboard/js/utils.js +23 -2
- package/templates/dashboard/js/view-live.js +235 -42
- package/templates/dashboard/js/view-runs.js +469 -42
- package/templates/dashboard/js/view-tests.js +157 -16
- package/templates/dashboard/js/view-tools.js +234 -0
- package/templates/dashboard/js/view-watch.js +2 -2
- package/templates/dashboard/js/websocket.js +33 -3
- package/templates/dashboard/styles/base.css +489 -53
- package/templates/dashboard/styles/components.css +736 -84
- package/templates/dashboard/styles/view-live.css +459 -78
- package/templates/dashboard/styles/view-runs.css +826 -177
- package/templates/dashboard/styles/view-tests.css +440 -77
- package/templates/dashboard/styles/view-tools.css +206 -0
- package/templates/dashboard/styles/view-watch.css +198 -41
- package/templates/dashboard/template.html +356 -58
- package/templates/dashboard.html +5354 -722
- package/templates/docker-compose-lightpanda.yml +7 -0
package/README.md
CHANGED
|
@@ -21,91 +21,150 @@
|
|
|
21
21
|
<a href="https://skills.sh"><img src="https://img.shields.io/badge/skills.sh-e2e--testing-ff6600" alt="Agent Skills" /></a>
|
|
22
22
|
</p>
|
|
23
23
|
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
**E2E Runner** lets you test your web app without writing test code. Tests are plain JSON โ and you don't even have to write that yourself: **just ask Claude Code.**
|
|
27
|
+
|
|
28
|
+
## ๐ฌ Write a test by asking โ then watch it run
|
|
29
|
+
|
|
24
30
|
<p align="center">
|
|
25
|
-
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/
|
|
31
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/demo-live.gif" alt="Live dashboard streaming screenshots as a test suite runs" width="820" />
|
|
32
|
+
<br/><sub><em>The live dashboard while a suite runs โ every step streams a screenshot into the feed, in real time.</em></sub>
|
|
26
33
|
</p>
|
|
27
34
|
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
**E2E Runner** is a zero-code browser testing framework where tests are plain JSON files โ no Playwright scripts, no Cypress boilerplate, no test framework to learn. Define what to click, type, and assert, and the runner executes it in parallel against a shared Chrome pool.
|
|
35
|
+
With the built-in [MCP server](https://modelcontextprotocol.io/), creating a test is a conversation โ no docs, no syntax to memorize:
|
|
31
36
|
|
|
32
|
-
|
|
37
|
+
> **You:** *Create an E2E test for the login flow and run it.*
|
|
38
|
+
>
|
|
39
|
+
> **Claude Code:** *writes the test, runs it in a real browser, and reports back โ*
|
|
40
|
+
> โ
`login-flow` passed in 2.3s ยท screenshot saved ยท no network errors.
|
|
33
41
|
|
|
34
|
-
|
|
42
|
+
Behind the scenes Claude just wrote and ran this. A test is **just JSON** โ an ordered list of what a user does:
|
|
35
43
|
|
|
36
44
|
```json
|
|
37
45
|
[
|
|
38
|
-
{
|
|
39
|
-
"
|
|
40
|
-
"
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
{ "type": "screenshot", "value": "logged-in.png" }
|
|
47
|
-
]
|
|
48
|
-
}
|
|
46
|
+
{ "name": "login-flow", "actions": [
|
|
47
|
+
{ "type": "goto", "value": "/login" },
|
|
48
|
+
{ "type": "type", "selector": "#email", "value": "user@test.com" },
|
|
49
|
+
{ "type": "type", "selector": "#password", "value": "secret" },
|
|
50
|
+
{ "type": "click", "text": "Sign In" },
|
|
51
|
+
{ "type": "assert_text", "text": "Welcome back" },
|
|
52
|
+
{ "type": "screenshot", "value": "logged-in.png" }
|
|
53
|
+
]}
|
|
49
54
|
]
|
|
50
55
|
```
|
|
51
56
|
|
|
52
|
-
No imports
|
|
53
|
-
|
|
54
|
-
---
|
|
55
|
-
|
|
56
|
-
## Agent Skills
|
|
57
|
+
No imports, no `describe`/`it`, no build step. If you can read it you can write it โ or just ask.
|
|
57
58
|
|
|
58
|
-
|
|
59
|
+
**Connect it to Claude Code (2 commands):**
|
|
59
60
|
|
|
60
61
|
```bash
|
|
61
|
-
|
|
62
|
+
claude plugin marketplace add fastslack/mtw-e2e-runner
|
|
63
|
+
claude plugin install e2e-runner@matware
|
|
62
64
|
```
|
|
63
65
|
|
|
64
|
-
|
|
66
|
+
Now say *"create a test for X and run it"* โ Claude gets 17 MCP tools, slash commands, and specialized agents.
|
|
65
67
|
|
|
66
|
-
>
|
|
68
|
+
> Using a different agent (Cursor, Codex, Copilot, [40+ more](https://github.com/vercel-labs/skills#supported-agents))? Install the skill: `npx skills add fastslack/mtw-e2e-runner`
|
|
67
69
|
|
|
68
70
|
---
|
|
69
71
|
|
|
70
|
-
##
|
|
72
|
+
## ๐ Contents
|
|
73
|
+
|
|
74
|
+
| | Section | What's inside |
|
|
75
|
+
|---|---------|---------------|
|
|
76
|
+
| ๐ | **[Install & first test](#install)** | npm setup ยท run with your own Chrome (no Docker), Obscura, or a Docker pool |
|
|
77
|
+
| โจ | **[What you get](#features)** | feature overview at a glance |
|
|
78
|
+
| โ๏ธ | **[Writing tests](#writing-tests)** | test format ยท full action catalog ยท retries ยท serial ยท modules ยท auth ยท hooks |
|
|
79
|
+
| ๐ค | **[AI integration](#ai)** | Claude Code ยท OpenCode ยท 17 MCP tools ยท visual verification ยท issue-to-test |
|
|
80
|
+
| ๐ | **[Dashboard & insights](#dashboard)** | live dashboard ยท learning system ยท network logs ยท screenshot capture |
|
|
81
|
+
| ๐ | **[Browser drivers](#drivers)** | browserless ยท cdp ยท lightpanda ยท obscura ยท steel |
|
|
82
|
+
| โ๏ธ | **[CLI, config & CI](#reference)** | commands ยท flags ยท `e2e.config.js` ยท GitHub Actions ยท programmatic API |
|
|
83
|
+
|
|
84
|
+
---
|
|
71
85
|
|
|
72
|
-
|
|
86
|
+
<a name="install"></a>
|
|
73
87
|
|
|
74
|
-
|
|
88
|
+
## ๐ Install โ it's tiny
|
|
75
89
|
|
|
76
90
|
```bash
|
|
77
91
|
npm install --save-dev @matware/e2e-runner
|
|
78
|
-
npx e2e-runner init
|
|
79
|
-
npx e2e-runner pool start # starts Chrome in Docker
|
|
80
|
-
npx e2e-runner run --all # runs the sample test
|
|
92
|
+
npx e2e-runner init # scaffolds e2e/ with a sample test + config
|
|
81
93
|
```
|
|
82
94
|
|
|
83
|
-
|
|
95
|
+
Then pick how to run the browser. **You don't need Docker** unless you want the parallel pool:
|
|
96
|
+
|
|
97
|
+
### Option 1 ยท Use the Chrome you already have โ no Docker โญ
|
|
98
|
+
|
|
99
|
+
Launch any Chromium browser with a debugging port, then point the runner at it:
|
|
84
100
|
|
|
85
101
|
```bash
|
|
86
|
-
|
|
102
|
+
google-chrome --headless=new --remote-debugging-port=9222 & # or brave / chromium / msedge
|
|
103
|
+
CHROME_POOL_URL=http://localhost:9222 POOL_DRIVER=cdp npx e2e-runner run --all
|
|
87
104
|
```
|
|
88
105
|
|
|
89
|
-
|
|
106
|
+
Or bake it into `e2e.config.js` so you never repeat it:
|
|
90
107
|
|
|
91
108
|
```js
|
|
92
109
|
export default {
|
|
93
|
-
baseUrl: 'http://
|
|
110
|
+
baseUrl: 'http://localhost:3000', // your app โ plain localhost, no docker hostname
|
|
111
|
+
poolUrls: ['http://localhost:9222'],
|
|
112
|
+
poolDriver: 'cdp',
|
|
94
113
|
};
|
|
95
114
|
```
|
|
96
115
|
|
|
97
|
-
|
|
116
|
+
Nothing to install beyond npm, and `baseUrl` is just `localhost` (the browser is on your machine).
|
|
98
117
|
|
|
99
|
-
###
|
|
118
|
+
### Option 2 ยท Obscura โ one tiny binary, no Docker
|
|
119
|
+
|
|
120
|
+
A single ~30 MB binary with built-in anti-detection. Install once, run it, point the runner at it:
|
|
100
121
|
|
|
101
122
|
```bash
|
|
102
|
-
|
|
103
|
-
|
|
123
|
+
obscura serve --port 9222 --stealth &
|
|
124
|
+
CHROME_POOL_URL=http://localhost:9222 POOL_DRIVER=obscura npx e2e-runner run --all
|
|
104
125
|
```
|
|
105
126
|
|
|
106
|
-
|
|
127
|
+
`npx e2e-runner pool start` (with `poolDriver: 'obscura'` in your config) prints the exact install command for your OS.
|
|
107
128
|
|
|
108
|
-
###
|
|
129
|
+
### Option 3 ยท Docker pool โ parallel, for CI & big suites
|
|
130
|
+
|
|
131
|
+
A shared, queue-managed Chrome pool that runs many tests at once:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
npx e2e-runner run --all # the first run auto-starts the Docker pool for you
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Requires Docker. Set `baseUrl: 'http://host.docker.internal:3000'` so the containerized Chrome can reach your app.
|
|
138
|
+
|
|
139
|
+
<details>
|
|
140
|
+
<summary><strong>Why <code>host.docker.internal</code> (Docker option only)?</strong></summary>
|
|
141
|
+
|
|
142
|
+
<br/>
|
|
143
|
+
|
|
144
|
+
With the Docker pool, Chrome runs inside a container, so `localhost` there means the container โ not your machine. `host.docker.internal` bridges to your host. On Linux (Docker Engine, not Docker Desktop) add `--add-host=host.docker.internal:host-gateway`, or use your LAN IP. Options 1 & 2 don't have this โ the browser is local, so plain `localhost` just works.
|
|
145
|
+
|
|
146
|
+
</details>
|
|
147
|
+
|
|
148
|
+
### Write your first test
|
|
149
|
+
|
|
150
|
+
Open `e2e/tests/sample.json` โ a flow is an ordered list of actions:
|
|
151
|
+
|
|
152
|
+
```json
|
|
153
|
+
[
|
|
154
|
+
{ "name": "homepage loads", "actions": [
|
|
155
|
+
{ "type": "goto", "value": "/" },
|
|
156
|
+
{ "type": "assert_text", "text": "Welcome" },
|
|
157
|
+
{ "type": "screenshot", "value": "home.png" }
|
|
158
|
+
]}
|
|
159
|
+
]
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
Run it with `npx e2e-runner run --all`. Results โ pass/fail, timing, screenshots, network errors โ print to your terminal and to the [web dashboard](#dashboard) if it's open.
|
|
163
|
+
|
|
164
|
+
<details>
|
|
165
|
+
<summary><strong>Add OpenCode</strong> (optional)</summary>
|
|
166
|
+
|
|
167
|
+
<br/>
|
|
109
168
|
|
|
110
169
|
```bash
|
|
111
170
|
cp node_modules/@matware/e2e-runner/opencode.json ./
|
|
@@ -114,21 +173,36 @@ mkdir -p .opencode && cp -r node_modules/@matware/e2e-runner/.opencode/* .openco
|
|
|
114
173
|
|
|
115
174
|
See [OPENCODE.md](OPENCODE.md) for details.
|
|
116
175
|
|
|
117
|
-
|
|
176
|
+
</details>
|
|
177
|
+
|
|
178
|
+
### Updating
|
|
179
|
+
|
|
180
|
+
Each install method updates separately โ bump the one(s) you use:
|
|
118
181
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
182
|
+
```bash
|
|
183
|
+
# npm dependency (per project)
|
|
184
|
+
npm install --save-dev @matware/e2e-runner@latest
|
|
185
|
+
|
|
186
|
+
# Claude Code plugin
|
|
187
|
+
claude plugin update e2e-runner@matware
|
|
188
|
+
|
|
189
|
+
# MCP-only install (npx caches the package โ pin @latest to force a refresh)
|
|
190
|
+
claude mcp add --transport stdio --scope user e2e-runner \
|
|
191
|
+
-- npx -y -p @matware/e2e-runner@latest e2e-runner-mcp
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
> [!NOTE]
|
|
195
|
+
> Two gotchas: **(1)** `npx` prefers a copy found in the project's `node_modules` over its own cache โ if a project pins an old version, the MCP server and dashboard run that old version, so update the project dependency too. **(2)** Already-running processes keep the old code in memory: after updating, restart the dashboard and reconnect the MCP server (`/mcp` โ `e2e-runner` โ Reconnect, or restart your session).
|
|
124
196
|
|
|
125
197
|
---
|
|
126
198
|
|
|
127
|
-
|
|
199
|
+
<a name="features"></a>
|
|
200
|
+
|
|
201
|
+
## โจ What you get
|
|
128
202
|
|
|
129
203
|
๐งช **Zero-code tests** โ JSON files that anyone on your team can read and write. No JavaScript, no compilation, no framework lock-in.
|
|
130
204
|
|
|
131
|
-
๐ค **AI-powered testing** โ Claude Code creates, executes, and debugs tests natively through
|
|
205
|
+
๐ค **AI-powered testing** โ Claude Code creates, executes, and debugs tests natively through 17 MCP tools. Ask it to "test the checkout flow" and it builds the JSON, runs it, and reports back.
|
|
132
206
|
|
|
133
207
|
๐ **Issue-to-Test pipeline** โ Paste a GitHub or GitLab issue URL. The runner fetches it, generates E2E tests, runs them, and tells you: *bug confirmed* or *not reproducible*.
|
|
134
208
|
|
|
@@ -136,7 +210,9 @@ See [OPENCODE.md](OPENCODE.md) for details.
|
|
|
136
210
|
|
|
137
211
|
๐ง **Learning system** โ Tracks test stability across runs. Detects flaky tests, unstable selectors, slow APIs, and error patterns โ then surfaces actionable insights.
|
|
138
212
|
|
|
139
|
-
โก **Parallel execution** โ Run N tests simultaneously against a shared
|
|
213
|
+
โก **Parallel execution** โ Run N tests simultaneously against a shared browser pool (browserless, raw CDP, Lightpanda, Obscura, or Steel). Serial mode available for tests that share state.
|
|
214
|
+
|
|
215
|
+
๐ฏ **Pluggable browser drivers** โ Pick the engine that fits each test: real Chrome via browserless, Lightpanda or Obscura for fast lightweight runs, Steel for managed sessions. Set `driver` per test or override the whole run with `--driver`.
|
|
140
216
|
|
|
141
217
|
๐ **Real-time dashboard** โ Live execution view, run history with pass-rate charts, screenshot gallery with hash-based search, expandable network request logs.
|
|
142
218
|
|
|
@@ -152,7 +228,16 @@ See [OPENCODE.md](OPENCODE.md) for details.
|
|
|
152
228
|
|
|
153
229
|
---
|
|
154
230
|
|
|
155
|
-
|
|
231
|
+
<a name="writing-tests"></a>
|
|
232
|
+
|
|
233
|
+
## โ๏ธ Writing tests
|
|
234
|
+
|
|
235
|
+
Everything about authoring tests โ the file format, the full action vocabulary, retries, state isolation, and reuse. Expand what you need:
|
|
236
|
+
|
|
237
|
+
<details>
|
|
238
|
+
<summary><strong>Test format & file layout</strong></summary>
|
|
239
|
+
|
|
240
|
+
<br/>
|
|
156
241
|
|
|
157
242
|
Each `.json` file in `e2e/tests/` contains an array of tests. Each test has a `name` and sequential `actions`:
|
|
158
243
|
|
|
@@ -172,14 +257,19 @@ Each `.json` file in `e2e/tests/` contains an array of tests. Each test has a `n
|
|
|
172
257
|
|
|
173
258
|
Suite files can have numeric prefixes for ordering (`01-auth.json`, `02-dashboard.json`). The `--suite` flag matches with or without the prefix, so `--suite auth` finds `01-auth.json`.
|
|
174
259
|
|
|
175
|
-
|
|
260
|
+
</details>
|
|
261
|
+
|
|
262
|
+
<details>
|
|
263
|
+
<summary><strong>Action catalog</strong> โ navigation, input & interaction</summary>
|
|
264
|
+
|
|
265
|
+
<br/>
|
|
176
266
|
|
|
177
267
|
| Action | Fields | Description |
|
|
178
268
|
|--------|--------|-------------|
|
|
179
269
|
| `goto` | `value` | Navigate to URL (relative to `baseUrl` or absolute) |
|
|
180
|
-
| `click` | `selector` or `text` | Click by CSS selector or visible text content |
|
|
270
|
+
| `click` | `selector` or `text` | Click by CSS selector or visible text content. Text mode also takes `scope: "dialog"`, `visible: true`, `last: true` |
|
|
181
271
|
| `type` / `fill` | `selector`, `value` | Clear field and type text |
|
|
182
|
-
| `wait` | `selector`, `text`, or `value` (ms) | Wait for element,
|
|
272
|
+
| `wait` | `selector`, `text`, `gone`, or `value` (ms) | Wait for element/text to appear, for `gone` to disappear (spinner/dialog), or fixed delay. Prefer conditions over fixed `value` sleeps |
|
|
183
273
|
| `screenshot` | `value` (filename) | Capture a screenshot |
|
|
184
274
|
| `select` | `selector`, `value` | Select a dropdown option |
|
|
185
275
|
| `clear` | `selector` | Clear an input field |
|
|
@@ -189,12 +279,33 @@ Suite files can have numeric prefixes for ordering (`01-auth.json`, `02-dashboar
|
|
|
189
279
|
| `evaluate` | `value` | Execute JavaScript in the browser context |
|
|
190
280
|
| `navigate` | `value` | Browser navigation (`back`, `forward`, `reload`) |
|
|
191
281
|
| `clear_cookies` | โ | Clear all cookies for the current page |
|
|
282
|
+
| `wait_network_idle` | optional `value` (idle ms, default 500), `timeout` | Wait until the network has been idle for `value` ms โ useful after actions that trigger background requests |
|
|
283
|
+
| `set_storage` | `value` (`"key=val"`), optional `selector: "session"` | Set a `localStorage` key (or `sessionStorage` with `selector: "session"`) |
|
|
284
|
+
| `gql` | `value` (query), optional `text` (variables JSON), optional `selector` (assertion) | Run a GraphQL query/mutation via in-page `fetch`, with the auth token read from `localStorage`. Fails on GraphQL errors. `selector` is a JS expression asserted against the response `r` (e.g. `"r.data.users.length > 0"`). Installs `window.__e2eGql` for later `evaluate` steps |
|
|
192
285
|
|
|
193
|
-
|
|
286
|
+
**Click by text** โ when `click` uses `text` instead of `selector`, it searches across common interactive and content elements:
|
|
287
|
+
|
|
288
|
+
```
|
|
289
|
+
button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"],
|
|
290
|
+
[role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
```json
|
|
294
|
+
{ "type": "click", "text": "Sign In" }
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
</details>
|
|
298
|
+
|
|
299
|
+
<details>
|
|
300
|
+
<summary><strong>Assertions</strong> โ verify text, elements, URLs, counts & network</summary>
|
|
301
|
+
|
|
302
|
+
<br/>
|
|
194
303
|
|
|
195
304
|
| Action | Fields | Description |
|
|
196
305
|
|--------|--------|-------------|
|
|
197
306
|
| `assert_text` | `text` | Assert text exists anywhere on the page (substring) |
|
|
307
|
+
| `assert_no_text` | `text` | Assert text does NOT appear anywhere on the page โ opposite of `assert_text` |
|
|
308
|
+
| `assert_text_in` | `selector`, `text`, optional `value: "exact"` | Assert text inside a scoped container. `text` is a case-insensitive regex by default; `value: "exact"` switches to case-sensitive substring |
|
|
198
309
|
| `assert_element_text` | `selector`, `text`, optional `value: "exact"` | Assert element's text contains (or exactly matches) the expected text |
|
|
199
310
|
| `assert_url` | `value` | Assert current URL path or full URL. Paths (`/dashboard`) compare against pathname only |
|
|
200
311
|
| `assert_visible` | `selector` | Assert element exists and is visible |
|
|
@@ -205,32 +316,30 @@ Suite files can have numeric prefixes for ordering (`01-auth.json`, `02-dashboar
|
|
|
205
316
|
| `assert_matches` | `selector`, `value` (regex) | Assert element text matches a regex pattern |
|
|
206
317
|
| `assert_count` | `selector`, `value` | Assert element count: exact (`"5"`), or operators (`">3"`, `">=1"`, `"<10"`) |
|
|
207
318
|
| `assert_no_network_errors` | โ | Fail if any network requests failed (e.g. `ERR_CONNECTION_REFUSED`) |
|
|
319
|
+
| `assert_storage` | `value` (`"key"` or `"key=expected"`), optional `selector: "session"` | Assert a `localStorage`/`sessionStorage` key exists or has a specific value |
|
|
320
|
+
| `assert_visual` | `value` (golden image), optional `selector`, `text` (max diff, e.g. `"0.02"`), `fullPage`, `maskRegions`, `threshold` | Visual regression: compare a screenshot against a golden reference. The first run saves the golden; later runs fail if more pixels differ than the threshold (default 2%) and write a diff image |
|
|
208
321
|
| `get_text` | `selector` | Extract element text (non-assertion, never fails). Result: `{ value: "..." }` |
|
|
209
322
|
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
When `click` uses `text` instead of `selector`, it searches across common interactive and content elements:
|
|
213
|
-
|
|
214
|
-
```
|
|
215
|
-
button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"],
|
|
216
|
-
[role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6
|
|
217
|
-
```
|
|
323
|
+
</details>
|
|
218
324
|
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
```
|
|
325
|
+
<details>
|
|
326
|
+
<summary><strong>Framework-aware actions</strong> โ React/MUI without <code>evaluate</code> boilerplate</summary>
|
|
222
327
|
|
|
223
|
-
|
|
328
|
+
<br/>
|
|
224
329
|
|
|
225
330
|
These actions handle common patterns in React/MUI apps that normally require verbose `evaluate` boilerplate:
|
|
226
331
|
|
|
227
332
|
| Action | Fields | Description |
|
|
228
333
|
|--------|--------|-------------|
|
|
229
|
-
| `type_react` | `selector`, `value` | Type into React controlled inputs using the native value setter. Dispatches `input` + `change` events so React state updates correctly. |
|
|
334
|
+
| `type_react` | `selector`, `value`, optional `blur`, `waitAfter` | Type into React controlled inputs using the native value setter. Dispatches `input` + `change` events so React state updates correctly. `blur: true` commits on blur; `waitAfter: "<ms>"` waits after (debounced autocomplete). |
|
|
230
335
|
| `click_regex` | `text` (regex), optional `selector`, optional `value: "last"` | Click element whose textContent matches a regex (case-insensitive). Default: first match. Use `value: "last"` for last match. |
|
|
231
336
|
| `click_option` | `text` | Click a `[role="option"]` element by text โ common in autocomplete/select dropdowns. |
|
|
337
|
+
| `select_combobox` | `text`, optional `selector`, `filter`, `openWait`/`filterWait`/`waitAfter` | Open a MUI Autocomplete/Select, optionally type `filter`, then click the option matching `text`. Falls back across `[role="option"]`, `.MuiAutocomplete-option`, `li.MuiMenuItem-root`. |
|
|
232
338
|
| `focus_autocomplete` | `text` (label text) | Focus an autocomplete input by its label text. Supports MUI and generic `[role="combobox"]`. |
|
|
233
339
|
| `click_chip` | `text` | Click a chip/tag element by text. Searches `[class*="Chip"]`, `[class*="chip"]`, `[data-chip]`. |
|
|
340
|
+
| `click_icon` | `value` (icon id), optional `selector` (scope) | Click an icon by `data-testid`/`data-icon`/`aria-label`/class fragment or SVG `<title>` โ MUI, FontAwesome, Heroicons, etc. Clicks the nearest clickable ancestor (button, link, tab). |
|
|
341
|
+
| `click_menu_item` | `text`, optional `selector` (scope) | Click a menu item by text across `[role="menuitem"]`, `.dropdown-item`, `.menu-item`, MUI `MenuItem`. |
|
|
342
|
+
| `click_in_context` | `text` (container text), `selector` (child) | Click a child element inside the smallest container matching `text` โ e.g. the delete button of one specific card/row. |
|
|
234
343
|
|
|
235
344
|
```json
|
|
236
345
|
// Before: 5 lines of evaluate boilerplate
|
|
@@ -240,13 +349,38 @@ These actions handle common patterns in React/MUI apps that normally require ver
|
|
|
240
349
|
{ "type": "type_react", "selector": "#search", "value": "term" }
|
|
241
350
|
```
|
|
242
351
|
|
|
243
|
-
|
|
352
|
+
</details>
|
|
353
|
+
|
|
354
|
+
<details>
|
|
355
|
+
<summary><strong>Multi-tab actions</strong> โ popups, OAuth windows & cross-tab flows</summary>
|
|
356
|
+
|
|
357
|
+
<br/>
|
|
358
|
+
|
|
359
|
+
| Action | Fields | Description |
|
|
360
|
+
|--------|--------|-------------|
|
|
361
|
+
| `open_tab` | `value` (URL), optional `text` (label) | Open a new tab and navigate to the URL (relative to `baseUrl` or absolute). Label defaults to `tab-<n>` |
|
|
362
|
+
| `switch_tab` | `value` | Switch the active tab by label, numeric index, or title/URL match (regex or substring). `"default"` returns to the original tab |
|
|
363
|
+
| `wait_for_tab` | optional `text` (label), `timeout` | Wait for a new tab/popup opened by the app (`window.open`, `target="_blank"`) and make it the active tab |
|
|
364
|
+
| `assert_tab_count` | `value` | Assert the number of open tabs: exact (`"2"`) or operators (`">=2"`) |
|
|
365
|
+
| `close_tab` | optional `value` (label) | Close the current (or named) tab and switch back to the last remaining one |
|
|
366
|
+
|
|
367
|
+
All subsequent actions run in the active tab:
|
|
368
|
+
|
|
369
|
+
```json
|
|
370
|
+
{ "type": "click", "text": "Open report" }
|
|
371
|
+
{ "type": "wait_for_tab", "text": "report" }
|
|
372
|
+
{ "type": "assert_text", "text": "Quarterly results" }
|
|
373
|
+
{ "type": "close_tab" }
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
</details>
|
|
244
377
|
|
|
245
|
-
|
|
378
|
+
<details>
|
|
379
|
+
<summary><strong>Retries & flaky detection</strong></summary>
|
|
246
380
|
|
|
247
|
-
|
|
381
|
+
<br/>
|
|
248
382
|
|
|
249
|
-
|
|
383
|
+
**Test-level retry** โ retry an entire test on failure. Set globally via config or per-test:
|
|
250
384
|
|
|
251
385
|
```json
|
|
252
386
|
{ "name": "flaky-test", "retries": 3, "timeout": 15000, "actions": [...] }
|
|
@@ -254,9 +388,7 @@ Retry an entire test on failure. Set globally via config or per-test:
|
|
|
254
388
|
|
|
255
389
|
Tests that pass after retry are flagged as **flaky** in the report and learning system.
|
|
256
390
|
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
Retry a single action without rerunning the entire test. Useful for timing-sensitive clicks and waits:
|
|
391
|
+
**Action-level retry** โ retry a single action without rerunning the entire test. Useful for timing-sensitive clicks and waits:
|
|
260
392
|
|
|
261
393
|
```json
|
|
262
394
|
{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
|
|
@@ -265,9 +397,12 @@ Retry a single action without rerunning the entire test. Useful for timing-sensi
|
|
|
265
397
|
|
|
266
398
|
Set globally: `actionRetries` in config, `--action-retries <n>` CLI, or `ACTION_RETRIES` env var. Delay between retries: `actionRetryDelay` (default 500ms).
|
|
267
399
|
|
|
268
|
-
|
|
400
|
+
</details>
|
|
401
|
+
|
|
402
|
+
<details>
|
|
403
|
+
<summary><strong>Serial tests</strong> โ for tests that share state</summary>
|
|
269
404
|
|
|
270
|
-
|
|
405
|
+
<br/>
|
|
271
406
|
|
|
272
407
|
Tests that share state (e.g., two tests modifying the same record) can race when running in parallel. Mark them as serial:
|
|
273
408
|
|
|
@@ -278,9 +413,12 @@ Tests that share state (e.g., two tests modifying the same record) can race when
|
|
|
278
413
|
|
|
279
414
|
Serial tests run one at a time **after** all parallel tests finish โ preventing interference without slowing down independent tests.
|
|
280
415
|
|
|
281
|
-
|
|
416
|
+
</details>
|
|
417
|
+
|
|
418
|
+
<details>
|
|
419
|
+
<summary><strong>Testing authenticated apps</strong></summary>
|
|
282
420
|
|
|
283
|
-
|
|
421
|
+
<br/>
|
|
284
422
|
|
|
285
423
|
The simplest approach โ log in via the UI like a real user:
|
|
286
424
|
|
|
@@ -319,9 +457,12 @@ Each test runs in a **fresh browser context**, so auth state is automatically cl
|
|
|
319
457
|
|
|
320
458
|
> **More strategies:** Cookie-based auth, HTTP header injection, OAuth/SSO bypasses, reusable auth modules, and role-based testing โ see [docs/authentication.md](docs/authentication.md)
|
|
321
459
|
|
|
322
|
-
|
|
460
|
+
</details>
|
|
461
|
+
|
|
462
|
+
<details>
|
|
463
|
+
<summary><strong>Reusable modules</strong> โ extract common flows with <code>$use</code></summary>
|
|
323
464
|
|
|
324
|
-
|
|
465
|
+
<br/>
|
|
325
466
|
|
|
326
467
|
Extract common flows into parameterized modules:
|
|
327
468
|
|
|
@@ -358,9 +499,35 @@ Use in tests:
|
|
|
358
499
|
|
|
359
500
|
Modules support parameter validation (required params fail fast), conditional blocks (`{{#param}}...{{/param}}`), nested composition, and cycle detection.
|
|
360
501
|
|
|
361
|
-
|
|
502
|
+
</details>
|
|
362
503
|
|
|
363
|
-
|
|
504
|
+
<details>
|
|
505
|
+
<summary><strong>Hooks</strong> โ beforeAll / beforeEach / afterEach / afterAll</summary>
|
|
506
|
+
|
|
507
|
+
<br/>
|
|
508
|
+
|
|
509
|
+
Run actions at lifecycle points. Define globally in config or per-suite:
|
|
510
|
+
|
|
511
|
+
```json
|
|
512
|
+
{
|
|
513
|
+
"hooks": {
|
|
514
|
+
"beforeAll": [{ "type": "goto", "value": "/setup" }],
|
|
515
|
+
"beforeEach": [{ "type": "goto", "value": "/" }],
|
|
516
|
+
"afterEach": [{ "type": "screenshot", "value": "after.png" }],
|
|
517
|
+
"afterAll": []
|
|
518
|
+
},
|
|
519
|
+
"tests": [...]
|
|
520
|
+
}
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
> **Important:** `beforeAll` runs on a separate browser page that is closed before tests start. Use `beforeEach` for state that tests need (cookies, localStorage, auth tokens).
|
|
524
|
+
|
|
525
|
+
</details>
|
|
526
|
+
|
|
527
|
+
<details>
|
|
528
|
+
<summary><strong>Exclude patterns</strong> โ skip drafts from <code>--all</code></summary>
|
|
529
|
+
|
|
530
|
+
<br/>
|
|
364
531
|
|
|
365
532
|
Skip exploratory or draft tests from `--all` runs:
|
|
366
533
|
|
|
@@ -373,9 +540,84 @@ export default {
|
|
|
373
540
|
|
|
374
541
|
Individual suite runs (`--suite`) are not affected by exclude patterns.
|
|
375
542
|
|
|
543
|
+
</details>
|
|
544
|
+
|
|
376
545
|
---
|
|
377
546
|
|
|
378
|
-
|
|
547
|
+
<a name="ai"></a>
|
|
548
|
+
|
|
549
|
+
## ๐ค AI integration
|
|
550
|
+
|
|
551
|
+
The whole point: your agent writes, runs, and verifies tests for you.
|
|
552
|
+
|
|
553
|
+
<details>
|
|
554
|
+
<summary><strong>Claude Code</strong> โ plugin install & MCP-only install</summary>
|
|
555
|
+
|
|
556
|
+
<br/>
|
|
557
|
+
|
|
558
|
+
```bash
|
|
559
|
+
claude plugin marketplace add fastslack/mtw-e2e-runner
|
|
560
|
+
claude plugin install e2e-runner@matware
|
|
561
|
+
```
|
|
562
|
+
|
|
563
|
+
This gives Claude 17 MCP tools, a workflow skill, 4 slash commands (`/e2e-runner:run`, `/e2e-runner:create-test`, `/e2e-runner:verify-issue`, `/e2e-runner:capture`), and 3 specialized agents (test-analyzer, test-creator, test-improver).
|
|
564
|
+
|
|
565
|
+
**MCP-only install** (tools only, no skill/commands/agents):
|
|
566
|
+
|
|
567
|
+
```bash
|
|
568
|
+
claude mcp add --transport stdio --scope user e2e-runner \
|
|
569
|
+
-- npx -y -p @matware/e2e-runner e2e-runner-mcp
|
|
570
|
+
```
|
|
571
|
+
|
|
572
|
+
</details>
|
|
573
|
+
|
|
574
|
+
<details>
|
|
575
|
+
<summary><strong>OpenCode</strong></summary>
|
|
576
|
+
|
|
577
|
+
<br/>
|
|
578
|
+
|
|
579
|
+
```bash
|
|
580
|
+
cp node_modules/@matware/e2e-runner/opencode.json ./
|
|
581
|
+
mkdir -p .opencode && cp -r node_modules/@matware/e2e-runner/.opencode/* .opencode/
|
|
582
|
+
```
|
|
583
|
+
|
|
584
|
+
See [OPENCODE.md](OPENCODE.md) for details.
|
|
585
|
+
|
|
586
|
+
</details>
|
|
587
|
+
|
|
588
|
+
<details>
|
|
589
|
+
<summary><strong>The 17 MCP tools</strong></summary>
|
|
590
|
+
|
|
591
|
+
<br/>
|
|
592
|
+
|
|
593
|
+
| Tool | Description |
|
|
594
|
+
|------|-------------|
|
|
595
|
+
| `e2e_run` | Run tests (all, by suite, or by file) |
|
|
596
|
+
| `e2e_list` | List available test suites |
|
|
597
|
+
| `e2e_create_test` | Create a new test JSON file |
|
|
598
|
+
| `e2e_create_module` | Create a reusable module |
|
|
599
|
+
| `e2e_pool_status` | Check Chrome pool health |
|
|
600
|
+
| `e2e_app_pool_status` | Inspect the app environment pool (forks, ports, drivers) |
|
|
601
|
+
| `e2e_screenshot` | Retrieve a screenshot by hash |
|
|
602
|
+
| `e2e_capture` | Capture screenshot of any URL |
|
|
603
|
+
| `e2e_analyze` | Extract page structure (interactive elements, forms, headings) and emit test scaffolds |
|
|
604
|
+
| `e2e_dashboard_start` | Start web dashboard |
|
|
605
|
+
| `e2e_dashboard_stop` | Stop web dashboard |
|
|
606
|
+
| `e2e_dashboard_restart` | Restart the dashboard (new project dir/port, clear stale sessions) |
|
|
607
|
+
| `e2e_issue` | Fetch issue and generate tests |
|
|
608
|
+
| `e2e_network_logs` | Query network logs for a run |
|
|
609
|
+
| `e2e_learnings` | Query stability insights |
|
|
610
|
+
| `e2e_vars` | Manage SQLite-backed `{{var.KEY}}` project variables |
|
|
611
|
+
| `e2e_neo4j` | Manage Neo4j knowledge graph |
|
|
612
|
+
|
|
613
|
+
> Pool start/stop are CLI-only โ not exposed via MCP.
|
|
614
|
+
|
|
615
|
+
</details>
|
|
616
|
+
|
|
617
|
+
<details>
|
|
618
|
+
<summary><strong>Visual verification</strong> โ describe the page, AI judges it</summary>
|
|
619
|
+
|
|
620
|
+
<br/>
|
|
379
621
|
|
|
380
622
|
Describe what the page should look like โ AI judges pass/fail from screenshots:
|
|
381
623
|
|
|
@@ -392,9 +634,12 @@ Describe what the page should look like โ AI judges pass/fail from screenshots
|
|
|
392
634
|
|
|
393
635
|
After test actions complete, the runner auto-captures a verification screenshot. The MCP response includes the screenshot hash โ Claude Code retrieves it and visually verifies against your `expect` description. No API key required.
|
|
394
636
|
|
|
395
|
-
|
|
637
|
+
</details>
|
|
638
|
+
|
|
639
|
+
<details>
|
|
640
|
+
<summary><strong>Issue-to-test</strong> โ turn a bug report into a runnable test</summary>
|
|
396
641
|
|
|
397
|
-
|
|
642
|
+
<br/>
|
|
398
643
|
|
|
399
644
|
Turn GitHub and GitLab issues into executable E2E tests. Paste an issue URL and get runnable tests โ automatically.
|
|
400
645
|
|
|
@@ -423,164 +668,97 @@ In Claude Code, just ask:
|
|
|
423
668
|
|
|
424
669
|
**Auth:** GitHub requires `gh` CLI, GitLab requires `glab` CLI. Self-hosted GitLab is supported.
|
|
425
670
|
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
## Learning System
|
|
429
|
-
|
|
430
|
-
The runner learns from every test run โ building knowledge about your test suite over time.
|
|
431
|
-
|
|
432
|
-
Query insights via the `e2e_learnings` MCP tool:
|
|
433
|
-
|
|
434
|
-
| Query | Returns |
|
|
435
|
-
|-------|---------|
|
|
436
|
-
| `summary` | Full health overview: pass rate, flaky tests, unstable selectors, API issues |
|
|
437
|
-
| `flaky` | Tests that pass only after retries |
|
|
438
|
-
| `selectors` | CSS selectors with high failure rates |
|
|
439
|
-
| `pages` | Pages with console errors, network failures, load time issues |
|
|
440
|
-
| `apis` | API endpoints with error rates and latency (auto-normalized: UUIDs, hashes, IDs) |
|
|
441
|
-
| `errors` | Most frequent error patterns, categorized |
|
|
442
|
-
| `trends` | Pass rate over time (auto-switches to hourly when all data is from one day) |
|
|
443
|
-
| `test:<name>` | Drill-down history for a specific test |
|
|
444
|
-
| `page:<path>` | Drill-down history for a specific page |
|
|
445
|
-
| `selector:<value>` | Drill-down history for a specific selector |
|
|
446
|
-
|
|
447
|
-
**Storage & export:**
|
|
448
|
-
- SQLite (`~/.e2e-runner/dashboard.db`) โ default, zero setup
|
|
449
|
-
- Neo4j knowledge graph โ optional, for relationship-based analysis. Manage via `e2e_neo4j` MCP tool or `docker compose`
|
|
450
|
-
- Markdown report (`e2e/learnings.md`) โ auto-generated after each run
|
|
451
|
-
|
|
452
|
-
**Test narration:** Each test run generates a human-readable narrative of what happened step by step, visible in the CLI output and the dashboard.
|
|
671
|
+
</details>
|
|
453
672
|
|
|
454
673
|
---
|
|
455
674
|
|
|
456
|
-
|
|
675
|
+
<a name="dashboard"></a>
|
|
457
676
|
|
|
458
|
-
|
|
677
|
+
## ๐ Dashboard & insights
|
|
459
678
|
|
|
460
679
|
```bash
|
|
461
680
|
e2e-runner dashboard # Start on default port 8484
|
|
462
681
|
e2e-runner dashboard --port 9090 # Custom port
|
|
463
682
|
```
|
|
464
683
|
|
|
465
|
-
|
|
684
|
+
<details>
|
|
685
|
+
<summary><strong>Web dashboard tour</strong> โ live view, history, gallery, pool status</summary>
|
|
466
686
|
|
|
467
|
-
|
|
687
|
+
<br/>
|
|
688
|
+
|
|
689
|
+
**Live execution** โ monitor tests in real-time with step-by-step progress, durations, and active worker count.
|
|
468
690
|
|
|
469
691
|
<p align="center">
|
|
470
692
|
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-live-running.png" alt="Dashboard - Live test execution" width="800" />
|
|
471
693
|
</p>
|
|
472
694
|
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
Browse all test suites across multiple projects. Run a single suite or all tests with one click.
|
|
695
|
+
**Test suites** โ browse all suites across projects. Run a single suite or all tests with one click.
|
|
476
696
|
|
|
477
697
|
<p align="center">
|
|
478
698
|
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-suites.png" alt="Dashboard - Test suites grid" width="800" />
|
|
479
699
|
</p>
|
|
480
700
|
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
Track pass rate trends with the built-in chart. Click any row to expand full detail with per-test results, screenshot hashes, and errors.
|
|
701
|
+
**Run history** โ track pass-rate trends with the built-in chart. Click any row to expand full detail.
|
|
484
702
|
|
|
485
703
|
<p align="center">
|
|
486
704
|
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-runs.png" alt="Dashboard - Run history" width="800" />
|
|
487
705
|
</p>
|
|
488
706
|
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
Expanded view with PASS/FAIL badges, screenshot thumbnails with copyable hashes (`ss:77c28b5a`), formatted console errors, and network request logs.
|
|
707
|
+
**Run detail** โ PASS/FAIL badges, screenshot thumbnails with copyable hashes (`ss:77c28b5a`), formatted console errors, and network request logs.
|
|
492
708
|
|
|
493
709
|
<p align="center">
|
|
494
710
|
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-run-detail.png" alt="Dashboard - Run detail" width="800" />
|
|
495
711
|
</p>
|
|
496
712
|
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
Browse all captured screenshots with hash search. Includes action screenshots, error screenshots, and verification captures.
|
|
713
|
+
**Screenshot gallery** โ browse all captured screenshots with hash search (action, error, and verification captures).
|
|
500
714
|
|
|
501
715
|
<p align="center">
|
|
502
716
|
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-screenshots-gallery.png" alt="Dashboard - Screenshot gallery" width="800" />
|
|
503
717
|
</p>
|
|
504
718
|
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
Monitor Chrome pool health: available slots, running sessions, memory pressure.
|
|
719
|
+
**Pool status** โ Chrome pool health: available slots, running sessions, memory pressure.
|
|
508
720
|
|
|
509
721
|
<p align="center">
|
|
510
722
|
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-pool-status.png" alt="Dashboard - Pool status" width="800" />
|
|
511
723
|
</p>
|
|
512
724
|
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
## Screenshot Capture
|
|
725
|
+
</details>
|
|
516
726
|
|
|
517
|
-
|
|
727
|
+
<details>
|
|
728
|
+
<summary><strong>Learning system</strong> โ flaky tests, unstable selectors, slow APIs</summary>
|
|
518
729
|
|
|
519
|
-
|
|
520
|
-
e2e-runner capture https://example.com
|
|
521
|
-
e2e-runner capture https://example.com --full-page --selector ".loaded" --delay 2000
|
|
522
|
-
```
|
|
730
|
+
<br/>
|
|
523
731
|
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
Every screenshot gets a deterministic hash (`ss:a3f2b1c9`). Use `e2e_screenshot` to retrieve any screenshot by hash โ it returns the image with metadata (test name, step, type).
|
|
527
|
-
|
|
528
|
-
---
|
|
529
|
-
|
|
530
|
-
## AI Integration
|
|
531
|
-
|
|
532
|
-
### Claude Code
|
|
533
|
-
|
|
534
|
-
```bash
|
|
535
|
-
claude plugin marketplace add fastslack/mtw-e2e-runner
|
|
536
|
-
claude plugin install e2e-runner@matware
|
|
537
|
-
```
|
|
538
|
-
|
|
539
|
-
This gives Claude 13 MCP tools, a workflow skill, 3 slash commands (`/e2e-runner:run`, `/e2e-runner:create-test`, `/e2e-runner:verify-issue`), and 3 specialized agents (test-analyzer, test-creator, test-improver).
|
|
540
|
-
|
|
541
|
-
**MCP-only install** (tools only, no skill/commands/agents):
|
|
732
|
+
The runner learns from every test run โ building knowledge about your test suite over time. Query insights via the `e2e_learnings` MCP tool:
|
|
542
733
|
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
See [OPENCODE.md](OPENCODE.md) for details.
|
|
556
|
-
|
|
557
|
-
### MCP Tools
|
|
734
|
+
| Query | Returns |
|
|
735
|
+
|-------|---------|
|
|
736
|
+
| `summary` | Full health overview: pass rate, flaky tests, unstable selectors, API issues |
|
|
737
|
+
| `flaky` | Tests that pass only after retries |
|
|
738
|
+
| `selectors` | CSS selectors with high failure rates |
|
|
739
|
+
| `pages` | Pages with console errors, network failures, load time issues |
|
|
740
|
+
| `apis` | API endpoints with error rates and latency (auto-normalized: UUIDs, hashes, IDs) |
|
|
741
|
+
| `errors` | Most frequent error patterns, categorized |
|
|
742
|
+
| `trends` | Pass rate over time (auto-switches to hourly when all data is from one day) |
|
|
743
|
+
| `test:<name>` | Drill-down history for a specific test |
|
|
744
|
+
| `page:<path>` | Drill-down history for a specific page |
|
|
745
|
+
| `selector:<value>` | Drill-down history for a specific selector |
|
|
558
746
|
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
| `e2e_create_test` | Create a new test JSON file |
|
|
564
|
-
| `e2e_create_module` | Create a reusable module |
|
|
565
|
-
| `e2e_pool_status` | Check Chrome pool health |
|
|
566
|
-
| `e2e_screenshot` | Retrieve a screenshot by hash |
|
|
567
|
-
| `e2e_capture` | Capture screenshot of any URL |
|
|
568
|
-
| `e2e_dashboard_start` | Start web dashboard |
|
|
569
|
-
| `e2e_dashboard_stop` | Stop web dashboard |
|
|
570
|
-
| `e2e_issue` | Fetch issue and generate tests |
|
|
571
|
-
| `e2e_network_logs` | Query network logs for a run |
|
|
572
|
-
| `e2e_learnings` | Query stability insights |
|
|
573
|
-
| `e2e_neo4j` | Manage Neo4j knowledge graph |
|
|
747
|
+
**Storage & export:**
|
|
748
|
+
- SQLite (`~/.e2e-runner/dashboard.db`) โ default, zero setup
|
|
749
|
+
- Neo4j knowledge graph โ optional, for relationship-based analysis. Manage via `e2e_neo4j` MCP tool or `docker compose`
|
|
750
|
+
- Markdown report (`e2e/learnings.md`) โ auto-generated after each run
|
|
574
751
|
|
|
575
|
-
|
|
752
|
+
**Test narration:** Each test run generates a human-readable narrative of what happened step by step, visible in the CLI output and the dashboard.
|
|
576
753
|
|
|
577
|
-
|
|
754
|
+
</details>
|
|
578
755
|
|
|
579
|
-
|
|
756
|
+
<details>
|
|
757
|
+
<summary><strong>Network error handling</strong> โ assertions, global flag, full logging</summary>
|
|
580
758
|
|
|
581
|
-
|
|
759
|
+
<br/>
|
|
582
760
|
|
|
583
|
-
|
|
761
|
+
**Explicit assertion** โ place `assert_no_network_errors` after critical page loads:
|
|
584
762
|
|
|
585
763
|
```json
|
|
586
764
|
{ "type": "goto", "value": "/dashboard" },
|
|
@@ -588,9 +766,7 @@ Place `assert_no_network_errors` after critical page loads:
|
|
|
588
766
|
{ "type": "assert_no_network_errors" }
|
|
589
767
|
```
|
|
590
768
|
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
Set `failOnNetworkError: true` to automatically fail any test with network errors:
|
|
769
|
+
**Global flag** โ set `failOnNetworkError: true` to automatically fail any test with network errors:
|
|
594
770
|
|
|
595
771
|
```bash
|
|
596
772
|
e2e-runner run --all --fail-on-network-error
|
|
@@ -598,11 +774,9 @@ e2e-runner run --all --fail-on-network-error
|
|
|
598
774
|
|
|
599
775
|
When disabled (default), the runner still collects and reports network errors โ the MCP response includes a warning when tests pass but have network errors.
|
|
600
776
|
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
All XHR/fetch requests are captured with: URL, method, status, duration, request/response headers, and response body (truncated at 50KB). Viewable in the dashboard with expandable request detail rows.
|
|
777
|
+
**Full network logging** โ all XHR/fetch requests are captured with URL, method, status, duration, request/response headers, and response body (truncated at 50KB). Viewable in the dashboard with expandable request detail rows.
|
|
604
778
|
|
|
605
|
-
|
|
779
|
+
MCP drill-down flow:
|
|
606
780
|
|
|
607
781
|
```
|
|
608
782
|
1. e2e_run โ compact networkSummary + runDbId
|
|
@@ -614,29 +788,107 @@ All XHR/fetch requests are captured with: URL, method, status, duration, request
|
|
|
614
788
|
|
|
615
789
|
The `e2e_run` response stays compact (~5KB) regardless of how many requests were captured. Use `e2e_network_logs` with the returned `runDbId` to drill into details on demand.
|
|
616
790
|
|
|
791
|
+
</details>
|
|
792
|
+
|
|
793
|
+
<details>
|
|
794
|
+
<summary><strong>Screenshot capture</strong> โ snapshot any URL on demand</summary>
|
|
795
|
+
|
|
796
|
+
<br/>
|
|
797
|
+
|
|
798
|
+
Capture screenshots of any URL on demand โ no test suite required:
|
|
799
|
+
|
|
800
|
+
```bash
|
|
801
|
+
e2e-runner capture https://example.com
|
|
802
|
+
e2e-runner capture https://example.com --full-page --selector ".loaded" --delay 2000
|
|
803
|
+
```
|
|
804
|
+
|
|
805
|
+
Via MCP, the `e2e_capture` tool supports `authToken` and `authStorageKey` for authenticated pages โ it injects the token into localStorage before navigating.
|
|
806
|
+
|
|
807
|
+
Every screenshot gets a deterministic hash (`ss:a3f2b1c9`). Use `e2e_screenshot` to retrieve any screenshot by hash โ it returns the image with metadata (test name, step, type).
|
|
808
|
+
|
|
809
|
+
</details>
|
|
810
|
+
|
|
617
811
|
---
|
|
618
812
|
|
|
619
|
-
|
|
813
|
+
<a name="drivers"></a>
|
|
620
814
|
|
|
621
|
-
|
|
815
|
+
## ๐ Browser drivers
|
|
816
|
+
|
|
817
|
+
The runner can talk to multiple browser engines through different drivers. The default is **`auto`** โ it probes each pool URL and picks the right driver per pool.
|
|
818
|
+
|
|
819
|
+
| Driver | Engine | Detection probe | When to use |
|
|
820
|
+
|--------|--------|-----------------|-------------|
|
|
821
|
+
| `browserless` | Real Chromium via [browserless](https://www.browserless.io/) | `/pressure` returns JSON | Default. Production-grade JS execution, screencast, full Chrome behavior |
|
|
822
|
+
| `cdp` | Generic CDP-compatible (raw Chrome, etc.) | `/json/version` reachable | Fallback for any CDP server that isn't one of the others |
|
|
823
|
+
| `lightpanda` | [Lightpanda](https://lightpanda.io) (Zig) | `/json/version` Browser=lightpanda | ~9ร faster, ~16ร less memory than headless Chrome โ ideal for high-volume scrape-style tests |
|
|
824
|
+
| `obscura` | [Obscura](https://github.com/h4ckf0r0day/obscura) (Rust + V8) | `/json/version` Browser=obscura | ~30 MB RAM footprint, built-in anti-detection (`--stealth`), stays close to real Chrome via Puppeteer |
|
|
825
|
+
| `steel` | [Steel Browser](https://steel.dev) | `/v1/sessions` returns JSON | Managed session lifecycle, REST API for orchestration |
|
|
826
|
+
|
|
827
|
+
<details>
|
|
828
|
+
<summary><strong>Pick a driver per test / force one per run</strong></summary>
|
|
829
|
+
|
|
830
|
+
<br/>
|
|
622
831
|
|
|
623
832
|
```json
|
|
624
833
|
{
|
|
625
|
-
"
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
834
|
+
"tests": [
|
|
835
|
+
{
|
|
836
|
+
"name": "checkout flow (heavy JS, real Chrome)",
|
|
837
|
+
"driver": "browserless",
|
|
838
|
+
"actions": [...]
|
|
839
|
+
},
|
|
840
|
+
{
|
|
841
|
+
"name": "scrape product page (lightweight)",
|
|
842
|
+
"driver": "obscura",
|
|
843
|
+
"fallbackDriver": "cdp",
|
|
844
|
+
"actions": [...]
|
|
845
|
+
}
|
|
846
|
+
]
|
|
632
847
|
}
|
|
633
848
|
```
|
|
634
849
|
|
|
635
|
-
|
|
850
|
+
`driver` is optional. If set, only pools whose detected driver matches become candidates. `fallbackDriver` is **explicit opt-in** โ without it, a missing target driver fails the test with a clear message. Pool busyness does **not** trigger fallback; the runner waits inside the filtered set.
|
|
851
|
+
|
|
852
|
+
Force a driver for a whole run (CLI overrides win over per-test fields โ useful for A/B benchmarks):
|
|
853
|
+
|
|
854
|
+
```bash
|
|
855
|
+
e2e-runner run --all --driver obscura
|
|
856
|
+
e2e-runner run --all --driver obscura --fallback-driver cdp
|
|
857
|
+
```
|
|
858
|
+
|
|
859
|
+
</details>
|
|
860
|
+
|
|
861
|
+
<details>
|
|
862
|
+
<summary><strong>Running each driver locally</strong></summary>
|
|
863
|
+
|
|
864
|
+
<br/>
|
|
865
|
+
|
|
866
|
+
```bash
|
|
867
|
+
# browserless (default) โ managed by `pool start`
|
|
868
|
+
e2e-runner pool start
|
|
869
|
+
|
|
870
|
+
# Lightpanda โ pool start uses templates/docker-compose-lightpanda.yml
|
|
871
|
+
e2e-runner pool start # with poolDriver: 'lightpanda' in config
|
|
872
|
+
|
|
873
|
+
# Obscura โ install the binary and run it yourself
|
|
874
|
+
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
|
|
875
|
+
tar xzf obscura-x86_64-linux.tar.gz
|
|
876
|
+
./obscura serve --port 9222 --stealth
|
|
877
|
+
# then point the runner at it: poolUrls: ['http://localhost:9222'], poolDriver: 'obscura'
|
|
878
|
+
```
|
|
879
|
+
|
|
880
|
+
</details>
|
|
636
881
|
|
|
637
882
|
---
|
|
638
883
|
|
|
639
|
-
|
|
884
|
+
<a name="reference"></a>
|
|
885
|
+
|
|
886
|
+
## โ๏ธ CLI, config & CI
|
|
887
|
+
|
|
888
|
+
<details>
|
|
889
|
+
<summary><strong>CLI commands</strong></summary>
|
|
890
|
+
|
|
891
|
+
<br/>
|
|
640
892
|
|
|
641
893
|
```bash
|
|
642
894
|
# Run tests
|
|
@@ -664,7 +916,12 @@ e2e-runner capture <url> # On-demand screenshot
|
|
|
664
916
|
e2e-runner init # Scaffold project
|
|
665
917
|
```
|
|
666
918
|
|
|
667
|
-
|
|
919
|
+
</details>
|
|
920
|
+
|
|
921
|
+
<details>
|
|
922
|
+
<summary><strong>CLI options</strong></summary>
|
|
923
|
+
|
|
924
|
+
<br/>
|
|
668
925
|
|
|
669
926
|
| Flag | Default | Description |
|
|
670
927
|
|------|---------|-------------|
|
|
@@ -679,10 +936,15 @@ e2e-runner init # Scaffold project
|
|
|
679
936
|
| `--env <name>` | `default` | Environment profile |
|
|
680
937
|
| `--fail-on-network-error` | `false` | Fail tests with network errors |
|
|
681
938
|
| `--project-name <name>` | dir name | Project display name |
|
|
939
|
+
| `--driver <name>` | _(per-test)_ | Force pool driver for the run: `browserless`, `cdp`, `lightpanda`, `obscura`, `steel` |
|
|
940
|
+
| `--fallback-driver <name>` | _none_ | Explicit fallback if no pool with `--driver` is reachable |
|
|
682
941
|
|
|
683
|
-
|
|
942
|
+
</details>
|
|
943
|
+
|
|
944
|
+
<details>
|
|
945
|
+
<summary><strong>Configuration</strong> โ <code>e2e.config.js</code> & priority</summary>
|
|
684
946
|
|
|
685
|
-
|
|
947
|
+
<br/>
|
|
686
948
|
|
|
687
949
|
Create `e2e.config.js` in your project root:
|
|
688
950
|
|
|
@@ -708,7 +970,7 @@ export default {
|
|
|
708
970
|
};
|
|
709
971
|
```
|
|
710
972
|
|
|
711
|
-
|
|
973
|
+
**Config priority (highest wins):**
|
|
712
974
|
|
|
713
975
|
1. CLI flags
|
|
714
976
|
2. Environment variables
|
|
@@ -717,18 +979,17 @@ export default {
|
|
|
717
979
|
|
|
718
980
|
When `--env <name>` is set, the matching profile overrides everything.
|
|
719
981
|
|
|
720
|
-
|
|
982
|
+
</details>
|
|
721
983
|
|
|
722
|
-
|
|
984
|
+
<details>
|
|
985
|
+
<summary><strong>CI/CD</strong> โ JUnit XML & GitHub Actions</summary>
|
|
723
986
|
|
|
724
|
-
|
|
987
|
+
<br/>
|
|
725
988
|
|
|
726
989
|
```bash
|
|
727
990
|
e2e-runner run --all --output junit
|
|
728
991
|
```
|
|
729
992
|
|
|
730
|
-
### GitHub Actions
|
|
731
|
-
|
|
732
993
|
```yaml
|
|
733
994
|
jobs:
|
|
734
995
|
e2e:
|
|
@@ -747,9 +1008,12 @@ jobs:
|
|
|
747
1008
|
report_paths: e2e/screenshots/junit.xml
|
|
748
1009
|
```
|
|
749
1010
|
|
|
750
|
-
|
|
1011
|
+
</details>
|
|
751
1012
|
|
|
752
|
-
|
|
1013
|
+
<details>
|
|
1014
|
+
<summary><strong>Programmatic API</strong></summary>
|
|
1015
|
+
|
|
1016
|
+
<br/>
|
|
753
1017
|
|
|
754
1018
|
```js
|
|
755
1019
|
import { createRunner } from '@matware/e2e-runner';
|
|
@@ -764,15 +1028,17 @@ const report = await runner.runTests([
|
|
|
764
1028
|
]);
|
|
765
1029
|
```
|
|
766
1030
|
|
|
1031
|
+
</details>
|
|
1032
|
+
|
|
767
1033
|
---
|
|
768
1034
|
|
|
769
1035
|
## Requirements
|
|
770
1036
|
|
|
771
1037
|
- **Node.js** >= 20
|
|
772
|
-
- **Docker**
|
|
1038
|
+
- **Docker** โ only for [Option 3](#install) (the parallel Chrome pool). Options 1 & 2 don't need it.
|
|
773
1039
|
|
|
774
1040
|
## License
|
|
775
1041
|
|
|
776
|
-
Copyright
|
|
1042
|
+
Copyright 2026 Matias Aguirre (fastslack) โ Matware
|
|
777
1043
|
|
|
778
1044
|
Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.
|