@matware/e2e-runner 1.1.0 → 1.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +9 -0
- package/.mcp.json +9 -0
- package/README.md +505 -279
- package/agents/test-analyzer.md +81 -0
- package/agents/test-creator.md +102 -0
- package/agents/test-improver.md +140 -0
- package/bin/cli.js +275 -7
- package/commands/create-test.md +50 -0
- package/commands/run.md +49 -0
- package/commands/verify-issue.md +63 -0
- package/package.json +11 -3
- package/skills/e2e-testing/SKILL.md +166 -0
- package/skills/e2e-testing/references/action-types.md +100 -0
- package/skills/e2e-testing/references/test-json-format.md +159 -0
- package/skills/e2e-testing/references/troubleshooting.md +182 -0
- package/src/actions.js +280 -17
- package/src/ai-generate.js +122 -11
- package/src/config.js +58 -0
- package/src/dashboard.js +173 -10
- package/src/db.js +232 -17
- package/src/index.js +9 -3
- package/src/learner-markdown.js +177 -0
- package/src/learner-neo4j.js +255 -0
- package/src/learner-sqlite.js +354 -0
- package/src/learner.js +413 -0
- package/src/mcp-tools.js +575 -16
- package/src/module-resolver.js +273 -0
- package/src/narrate.js +225 -0
- package/src/neo4j-pool.js +124 -0
- package/src/reporter.js +47 -2
- package/src/runner.js +180 -40
- package/src/verify.js +19 -5
- package/templates/build-dashboard.js +28 -0
- package/templates/dashboard/app.js +1152 -0
- package/templates/dashboard/styles.css +413 -0
- package/templates/dashboard/template.html +201 -0
- package/templates/dashboard.html +1091 -268
- package/templates/docker-compose-neo4j.yml +19 -0
- package/templates/e2e.config.js +3 -0
package/README.md
CHANGED
|
@@ -1,13 +1,58 @@
|
|
|
1
|
+
<p align="right">
|
|
2
|
+
<strong>English</strong> · <a href="LEEME.md">Español</a>
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<h1 align="center">@matware/e2e-runner</h1>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
<strong>The AI-native E2E test runner that writes, runs, and debugs tests for you.</strong>
|
|
9
|
+
</p>
|
|
10
|
+
|
|
1
11
|
<p align="center">
|
|
2
12
|
<img src="https://img.shields.io/npm/v/@matware/e2e-runner?color=blue" alt="npm version" />
|
|
3
13
|
<img src="https://img.shields.io/node/v/@matware/e2e-runner" alt="node version" />
|
|
4
14
|
<img src="https://img.shields.io/npm/l/@matware/e2e-runner" alt="license" />
|
|
5
15
|
<img src="https://img.shields.io/badge/MCP-compatible-green" alt="MCP compatible" />
|
|
16
|
+
<img src="https://img.shields.io/badge/AI--native-Claude%20Code-blueviolet" alt="AI native" />
|
|
6
17
|
</p>
|
|
7
18
|
|
|
8
|
-
|
|
19
|
+
<p align="center">
|
|
20
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-live-running.png" alt="E2E Runner Dashboard - Live Execution" width="800" />
|
|
21
|
+
</p>
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
**E2E Runner** is a zero-code browser testing framework where tests are plain JSON files — no Playwright scripts, no Cypress boilerplate, no test framework to learn. Define what to click, type, and assert, and the runner executes it in parallel against a shared Chrome pool.
|
|
26
|
+
|
|
27
|
+
But what makes it truly different is its **deep AI integration**. With a built-in [MCP server](https://modelcontextprotocol.io/), Claude Code can create tests from a conversation, run them, read the results, capture screenshots, and even visually verify that pages look correct — all without leaving the chat. Paste a GitHub issue URL and get a runnable test back. That's the workflow.
|
|
28
|
+
|
|
29
|
+
### What you get
|
|
30
|
+
|
|
31
|
+
🧪 **Zero-code tests** — JSON files that anyone on your team can read and write. No JavaScript, no compilation, no framework lock-in.
|
|
32
|
+
|
|
33
|
+
🤖 **AI-powered testing** — Claude Code creates, executes, and debugs tests natively through 13 MCP tools. Ask it to "test the checkout flow" and it builds the JSON, runs it, and reports back.
|
|
34
|
+
|
|
35
|
+
🐛 **Issue-to-Test pipeline** — Paste a GitHub or GitLab issue URL. The runner fetches it, generates E2E tests, runs them, and tells you: *bug confirmed* or *not reproducible*.
|
|
36
|
+
|
|
37
|
+
👁️ **Visual verification** — Describe what the page should look like in plain English. The AI captures a screenshot and judges pass/fail against your description. No pixel-diffing setup needed.
|
|
38
|
+
|
|
39
|
+
🧠 **Learning system** — Tracks test stability across runs. Detects flaky tests, unstable selectors, slow APIs, and error patterns — then surfaces actionable insights.
|
|
40
|
+
|
|
41
|
+
⚡ **Parallel execution** — Run N tests simultaneously against a shared Chrome pool (browserless/chrome). Serial mode available for tests that share state.
|
|
42
|
+
|
|
43
|
+
📊 **Real-time dashboard** — Live execution view, run history with pass-rate charts, screenshot gallery with hash-based search, expandable network request logs.
|
|
44
|
+
|
|
45
|
+
🔁 **Smart retries** — Test-level and action-level retries with configurable delays. Flaky tests are detected and flagged automatically.
|
|
9
46
|
|
|
10
|
-
|
|
47
|
+
📦 **Reusable modules** — Extract common flows (login, navigation, setup) into parameterized modules and reference them with `$use`.
|
|
48
|
+
|
|
49
|
+
🏗️ **CI-ready** — JUnit XML output, exit code 1 on failure, auto-captured error screenshots. Drop-in GitHub Actions example included.
|
|
50
|
+
|
|
51
|
+
🌐 **Multi-project** — One dashboard aggregates test results from all your projects. One Chrome pool serves them all.
|
|
52
|
+
|
|
53
|
+
🐳 **Portable** — Chrome runs in Docker, tests are JSON files in your repo. Works on any machine with Node.js and Docker.
|
|
54
|
+
|
|
55
|
+
### This is a test
|
|
11
56
|
|
|
12
57
|
```json
|
|
13
58
|
[
|
|
@@ -25,42 +70,52 @@ JSON-driven E2E test runner. Define browser tests as simple JSON action arrays,
|
|
|
25
70
|
]
|
|
26
71
|
```
|
|
27
72
|
|
|
73
|
+
No imports. No `describe`/`it`. No compilation step. Just a JSON file that describes what a user does — and the runner makes it happen.
|
|
74
|
+
|
|
28
75
|
---
|
|
29
76
|
|
|
30
|
-
##
|
|
77
|
+
## Quick Start
|
|
31
78
|
|
|
32
|
-
-
|
|
33
|
-
- **Parallel** -- Run N tests simultaneously against a shared Chrome pool.
|
|
34
|
-
- **Portable** -- Chrome runs in Docker, tests run anywhere.
|
|
35
|
-
- **CI-ready** -- JUnit XML output, exit code 1 on failure, error screenshots.
|
|
36
|
-
- **AI-native** -- Built-in MCP server for Claude Code integration.
|
|
79
|
+
**One-liner** (requires Node.js >= 20 and Docker):
|
|
37
80
|
|
|
38
|
-
|
|
81
|
+
```bash
|
|
82
|
+
curl -fsSL https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/scripts/quickstart.sh | bash
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
**Step by step:**
|
|
39
86
|
|
|
40
87
|
```bash
|
|
41
|
-
# Install
|
|
42
|
-
npm install @matware/e2e-runner
|
|
88
|
+
# 1. Install
|
|
89
|
+
npm install --save-dev @matware/e2e-runner
|
|
43
90
|
|
|
44
|
-
# Scaffold project structure
|
|
91
|
+
# 2. Scaffold project structure
|
|
45
92
|
npx e2e-runner init
|
|
46
93
|
|
|
47
|
-
# Start Chrome pool (requires Docker)
|
|
94
|
+
# 3. Start Chrome pool (requires Docker)
|
|
48
95
|
npx e2e-runner pool start
|
|
49
96
|
|
|
50
|
-
# Run all tests
|
|
97
|
+
# 4. Run all tests
|
|
51
98
|
npx e2e-runner run --all
|
|
99
|
+
|
|
100
|
+
# 5. Open the dashboard
|
|
101
|
+
npx e2e-runner dashboard
|
|
52
102
|
```
|
|
53
103
|
|
|
54
|
-
|
|
104
|
+
**Add to Claude Code** (once, available in all projects):
|
|
55
105
|
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
e2e
|
|
106
|
+
```bash
|
|
107
|
+
# Full plugin: MCP tools + skills + commands + agents
|
|
108
|
+
claude plugin install npm:@matware/e2e-runner
|
|
109
|
+
|
|
110
|
+
# Or MCP-only (tools without skills/commands/agents):
|
|
111
|
+
claude mcp add --transport stdio --scope user e2e-runner \
|
|
112
|
+
-- npx -y -p @matware/e2e-runner e2e-runner-mcp
|
|
62
113
|
```
|
|
63
114
|
|
|
115
|
+
The **plugin** is the recommended approach — it installs the 13 MCP tools *plus* a skill that teaches Claude the optimal workflow, 3 slash commands (`/e2e-runner:run`, `/e2e-runner:create-test`, `/e2e-runner:verify-issue`), and 2 specialized agents for test analysis and creation.
|
|
116
|
+
|
|
117
|
+
---
|
|
118
|
+
|
|
64
119
|
## Test Format
|
|
65
120
|
|
|
66
121
|
Each `.json` file in `e2e/tests/` contains an array of tests. Each test has a `name` and sequential `actions`:
|
|
@@ -90,384 +145,565 @@ Suite files can have numeric prefixes for ordering (`01-auth.json`, `02-dashboar
|
|
|
90
145
|
| `click` | `selector` or `text` | Click by CSS selector or visible text content |
|
|
91
146
|
| `type` / `fill` | `selector`, `value` | Clear field and type text |
|
|
92
147
|
| `wait` | `selector`, `text`, or `value` (ms) | Wait for element, text, or fixed delay |
|
|
93
|
-
| `assert_text` | `text` | Assert text exists on the page |
|
|
94
|
-
| `assert_url` | `value` | Assert current URL contains value |
|
|
95
|
-
| `assert_visible` | `selector` | Assert element is visible |
|
|
96
|
-
| `assert_count` | `selector`, `value` | Assert element count matches |
|
|
97
148
|
| `screenshot` | `value` (filename) | Capture a screenshot |
|
|
98
149
|
| `select` | `selector`, `value` | Select a dropdown option |
|
|
99
150
|
| `clear` | `selector` | Clear an input field |
|
|
100
|
-
| `press` | `value` | Press a keyboard key (
|
|
151
|
+
| `press` | `value` | Press a keyboard key (`Enter`, `Tab`, etc.) |
|
|
101
152
|
| `scroll` | `selector` or `value` (px) | Scroll to element or by pixel amount |
|
|
102
153
|
| `hover` | `selector` | Hover over an element |
|
|
103
154
|
| `evaluate` | `value` | Execute JavaScript in the browser context |
|
|
155
|
+
| `navigate` | `value` | Browser navigation (`back`, `forward`, `reload`) |
|
|
156
|
+
| `clear_cookies` | — | Clear all cookies for the current page |
|
|
157
|
+
|
|
158
|
+
### Assertions
|
|
159
|
+
|
|
160
|
+
| Action | Fields | Description |
|
|
161
|
+
|--------|--------|-------------|
|
|
162
|
+
| `assert_text` | `text` | Assert text exists anywhere on the page (substring) |
|
|
163
|
+
| `assert_element_text` | `selector`, `text`, optional `value: "exact"` | Assert element's text contains (or exactly matches) the expected text |
|
|
164
|
+
| `assert_url` | `value` | Assert current URL path or full URL. Paths (`/dashboard`) compare against pathname only |
|
|
165
|
+
| `assert_visible` | `selector` | Assert element exists and is visible |
|
|
166
|
+
| `assert_not_visible` | `selector` | Assert element is hidden or doesn't exist |
|
|
167
|
+
| `assert_attribute` | `selector`, `value` | Check attribute: `"type=email"` for value, `"disabled"` for existence |
|
|
168
|
+
| `assert_class` | `selector`, `value` | Assert element has a CSS class |
|
|
169
|
+
| `assert_input_value` | `selector`, `value` | Assert input/select/textarea `.value` contains text |
|
|
170
|
+
| `assert_matches` | `selector`, `value` (regex) | Assert element text matches a regex pattern |
|
|
171
|
+
| `assert_count` | `selector`, `value` | Assert element count: exact (`"5"`), or operators (`">3"`, `">=1"`, `"<10"`) |
|
|
172
|
+
| `assert_no_network_errors` | — | Fail if any network requests failed (e.g. `ERR_CONNECTION_REFUSED`) |
|
|
173
|
+
| `get_text` | `selector` | Extract element text (non-assertion, never fails). Result: `{ value: "..." }` |
|
|
104
174
|
|
|
105
175
|
### Click by Text
|
|
106
176
|
|
|
107
|
-
When `click` uses `text` instead of `selector`, it searches across interactive elements:
|
|
177
|
+
When `click` uses `text` instead of `selector`, it searches across common interactive and content elements:
|
|
108
178
|
|
|
109
179
|
```
|
|
110
|
-
button, a, [role="button"], [role="tab"], [role="menuitem"],
|
|
180
|
+
button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"],
|
|
181
|
+
[role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6
|
|
111
182
|
```
|
|
112
183
|
|
|
113
184
|
```json
|
|
114
185
|
{ "type": "click", "text": "Sign In" }
|
|
115
186
|
```
|
|
116
187
|
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
```bash
|
|
120
|
-
# Run tests
|
|
121
|
-
npx e2e-runner run --all # All suites
|
|
122
|
-
npx e2e-runner run --suite auth # Single suite
|
|
123
|
-
npx e2e-runner run --tests path/to.json # Specific file
|
|
124
|
-
npx e2e-runner run --inline '<json>' # Inline JSON
|
|
188
|
+
### Framework-Aware Actions
|
|
125
189
|
|
|
126
|
-
|
|
127
|
-
npx e2e-runner pool start # Start Chrome container
|
|
128
|
-
npx e2e-runner pool stop # Stop Chrome container
|
|
129
|
-
npx e2e-runner pool status # Check pool health
|
|
190
|
+
These actions handle common patterns in React/MUI apps that normally require verbose `evaluate` boilerplate:
|
|
130
191
|
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
192
|
+
| Action | Fields | Description |
|
|
193
|
+
|--------|--------|-------------|
|
|
194
|
+
| `type_react` | `selector`, `value` | Type into React controlled inputs using the native value setter. Dispatches `input` + `change` events so React state updates correctly. |
|
|
195
|
+
| `click_regex` | `text` (regex), optional `selector`, optional `value: "last"` | Click element whose textContent matches a regex (case-insensitive). Default: first match. Use `value: "last"` for last match. |
|
|
196
|
+
| `click_option` | `text` | Click a `[role="option"]` element by text — common in autocomplete/select dropdowns. |
|
|
197
|
+
| `focus_autocomplete` | `text` (label text) | Focus an autocomplete input by its label text. Supports MUI and generic `[role="combobox"]`. |
|
|
198
|
+
| `click_chip` | `text` | Click a chip/tag element by text. Searches `[class*="Chip"]`, `[class*="chip"]`, `[data-chip]`. |
|
|
135
199
|
|
|
136
|
-
|
|
137
|
-
|
|
200
|
+
```json
|
|
201
|
+
// Before: 5 lines of evaluate boilerplate
|
|
202
|
+
{ "type": "evaluate", "value": "const input = document.querySelector('#search'); const nativeSet = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set; nativeSet.call(input, 'term'); input.dispatchEvent(new Event('input', {bubbles: true})); input.dispatchEvent(new Event('change', {bubbles: true}));" }
|
|
138
203
|
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
npx e2e-runner init # Scaffold project
|
|
204
|
+
// After: 1 action
|
|
205
|
+
{ "type": "type_react", "selector": "#search", "value": "term" }
|
|
142
206
|
```
|
|
143
207
|
|
|
144
|
-
|
|
208
|
+
---
|
|
145
209
|
|
|
146
|
-
|
|
147
|
-
|------|---------|-------------|
|
|
148
|
-
| `--base-url <url>` | `http://host.docker.internal:3000` | Application base URL |
|
|
149
|
-
| `--pool-url <ws>` | `ws://localhost:3333` | Chrome pool WebSocket URL |
|
|
150
|
-
| `--tests-dir <dir>` | `e2e/tests` | Tests directory |
|
|
151
|
-
| `--screenshots-dir <dir>` | `e2e/screenshots` | Screenshots/reports directory |
|
|
152
|
-
| `--concurrency <n>` | `3` | Parallel test workers |
|
|
153
|
-
| `--timeout <ms>` | `10000` | Default action timeout |
|
|
154
|
-
| `--retries <n>` | `0` | Retry failed tests N times |
|
|
155
|
-
| `--retry-delay <ms>` | `1000` | Delay between retries |
|
|
156
|
-
| `--test-timeout <ms>` | `60000` | Per-test timeout |
|
|
157
|
-
| `--output <format>` | `json` | Report format: `json`, `junit`, `both` |
|
|
158
|
-
| `--env <name>` | `default` | Environment profile |
|
|
159
|
-
| `--pool-port <port>` | `3333` | Chrome pool port |
|
|
160
|
-
| `--max-sessions <n>` | `10` | Max concurrent Chrome sessions |
|
|
161
|
-
| `--project-name <name>` | dir name | Project display name for dashboard |
|
|
210
|
+
## Retries
|
|
162
211
|
|
|
163
|
-
|
|
212
|
+
### Test-Level Retry
|
|
164
213
|
|
|
165
|
-
|
|
214
|
+
Retry an entire test on failure. Set globally via config or per-test:
|
|
166
215
|
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
concurrency: 4,
|
|
171
|
-
retries: 2,
|
|
172
|
-
testTimeout: 30000,
|
|
173
|
-
outputFormat: 'both',
|
|
216
|
+
```json
|
|
217
|
+
{ "name": "flaky-test", "retries": 3, "timeout": 15000, "actions": [...] }
|
|
218
|
+
```
|
|
174
219
|
|
|
175
|
-
|
|
176
|
-
beforeEach: [{ type: 'goto', value: '/' }],
|
|
177
|
-
afterEach: [{ type: 'screenshot', value: 'after-test.png' }],
|
|
178
|
-
},
|
|
220
|
+
Tests that pass after retry are flagged as **flaky** in the report and learning system.
|
|
179
221
|
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
222
|
+
### Action-Level Retry
|
|
223
|
+
|
|
224
|
+
Retry a single action without rerunning the entire test. Useful for timing-sensitive clicks and waits:
|
|
225
|
+
|
|
226
|
+
```json
|
|
227
|
+
{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
|
|
228
|
+
{ "type": "wait", "selector": ".lazy-loaded", "retries": 2 }
|
|
185
229
|
```
|
|
186
230
|
|
|
187
|
-
|
|
231
|
+
Set globally: `actionRetries` in config, `--action-retries <n>` CLI, or `ACTION_RETRIES` env var. Delay between retries: `actionRetryDelay` (default 500ms).
|
|
188
232
|
|
|
189
|
-
|
|
190
|
-
2. Environment variables (`BASE_URL`, `CONCURRENCY`, ...)
|
|
191
|
-
3. Config file (`e2e.config.js` or `e2e.config.json`)
|
|
192
|
-
4. Defaults
|
|
233
|
+
---
|
|
193
234
|
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
### Environment Variables
|
|
197
|
-
|
|
198
|
-
| Variable | Maps to |
|
|
199
|
-
|----------|---------|
|
|
200
|
-
| `BASE_URL` | `baseUrl` |
|
|
201
|
-
| `CHROME_POOL_URL` | `poolUrl` |
|
|
202
|
-
| `TESTS_DIR` | `testsDir` |
|
|
203
|
-
| `SCREENSHOTS_DIR` | `screenshotsDir` |
|
|
204
|
-
| `CONCURRENCY` | `concurrency` |
|
|
205
|
-
| `DEFAULT_TIMEOUT` | `defaultTimeout` |
|
|
206
|
-
| `POOL_PORT` | `poolPort` |
|
|
207
|
-
| `MAX_SESSIONS` | `maxSessions` |
|
|
208
|
-
| `RETRIES` | `retries` |
|
|
209
|
-
| `RETRY_DELAY` | `retryDelay` |
|
|
210
|
-
| `TEST_TIMEOUT` | `testTimeout` |
|
|
211
|
-
| `OUTPUT_FORMAT` | `outputFormat` |
|
|
212
|
-
| `E2E_ENV` | `env` |
|
|
213
|
-
| `PROJECT_NAME` | `projectName` |
|
|
214
|
-
| `ANTHROPIC_API_KEY` | `anthropicApiKey` |
|
|
215
|
-
| `ANTHROPIC_MODEL` | `anthropicModel` |
|
|
235
|
+
## Serial Tests
|
|
216
236
|
|
|
217
|
-
|
|
237
|
+
Tests that share state (e.g., two tests modifying the same record) can race when running in parallel. Mark them as serial:
|
|
218
238
|
|
|
219
|
-
|
|
239
|
+
```json
|
|
240
|
+
{ "name": "create-patient", "serial": true, "actions": [...] }
|
|
241
|
+
{ "name": "verify-patient-list", "serial": true, "actions": [...] }
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Serial tests run one at a time **after** all parallel tests finish — preventing interference without slowing down independent tests.
|
|
245
|
+
|
|
246
|
+
---
|
|
247
|
+
|
|
248
|
+
## Reusable Modules
|
|
249
|
+
|
|
250
|
+
Extract common flows into parameterized modules:
|
|
220
251
|
|
|
221
252
|
```json
|
|
253
|
+
// e2e/modules/auth.json
|
|
222
254
|
{
|
|
223
|
-
"
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
"
|
|
227
|
-
"
|
|
255
|
+
"$module": "auth-jwt",
|
|
256
|
+
"description": "Inject JWT token into localStorage",
|
|
257
|
+
"params": {
|
|
258
|
+
"token": { "required": true, "description": "JWT token" },
|
|
259
|
+
"storageKey": { "default": "accessToken" }
|
|
228
260
|
},
|
|
229
|
-
"
|
|
230
|
-
{ "
|
|
261
|
+
"actions": [
|
|
262
|
+
{ "type": "evaluate", "value": "localStorage.setItem('{{storageKey}}', '{{token}}')" },
|
|
263
|
+
{ "type": "goto", "value": "/dashboard" }
|
|
264
|
+
]
|
|
265
|
+
}
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
Use in tests:
|
|
269
|
+
|
|
270
|
+
```json
|
|
271
|
+
{
|
|
272
|
+
"name": "dashboard-loads",
|
|
273
|
+
"actions": [
|
|
274
|
+
{ "$use": "auth-jwt", "params": { "token": "eyJhbG..." } },
|
|
275
|
+
{ "type": "assert_text", "text": "Dashboard" }
|
|
231
276
|
]
|
|
232
277
|
}
|
|
233
278
|
```
|
|
234
279
|
|
|
235
|
-
|
|
280
|
+
Modules support parameter validation (required params fail fast), conditional blocks (`{{#param}}...{{/param}}`), nested composition, and cycle detection.
|
|
281
|
+
|
|
282
|
+
---
|
|
283
|
+
|
|
284
|
+
## Exclude Patterns
|
|
236
285
|
|
|
237
|
-
|
|
286
|
+
Skip exploratory or draft tests from `--all` runs:
|
|
287
|
+
|
|
288
|
+
```js
|
|
289
|
+
// e2e.config.js
|
|
290
|
+
export default {
|
|
291
|
+
exclude: ['explore-*', 'debug-*', 'draft-*'],
|
|
292
|
+
};
|
|
293
|
+
```
|
|
238
294
|
|
|
239
|
-
|
|
295
|
+
Individual suite runs (`--suite`) are not affected by exclude patterns.
|
|
296
|
+
|
|
297
|
+
---
|
|
298
|
+
|
|
299
|
+
## Visual Verification
|
|
300
|
+
|
|
301
|
+
Describe what the page should look like — AI judges pass/fail from screenshots:
|
|
240
302
|
|
|
241
303
|
```json
|
|
242
304
|
{
|
|
243
|
-
"name": "
|
|
244
|
-
"
|
|
245
|
-
"
|
|
246
|
-
|
|
305
|
+
"name": "dashboard-loads",
|
|
306
|
+
"expect": "Patient list with at least 3 rows, no error messages, sidebar with navigation links",
|
|
307
|
+
"actions": [
|
|
308
|
+
{ "type": "goto", "value": "/dashboard" },
|
|
309
|
+
{ "type": "wait", "selector": ".patient-list" }
|
|
310
|
+
]
|
|
247
311
|
}
|
|
248
312
|
```
|
|
249
313
|
|
|
250
|
-
|
|
251
|
-
- **Timeout**: Applied via `Promise.race()`. Defaults to 60s.
|
|
314
|
+
After test actions complete, the runner auto-captures a verification screenshot. The MCP response includes the screenshot hash — Claude Code retrieves it and visually verifies against your `expect` description. No API key required.
|
|
252
315
|
|
|
253
|
-
|
|
316
|
+
---
|
|
254
317
|
|
|
255
|
-
|
|
318
|
+
## Issue-to-Test
|
|
319
|
+
|
|
320
|
+
Turn GitHub and GitLab issues into executable E2E tests. Paste an issue URL and get runnable tests — automatically.
|
|
321
|
+
|
|
322
|
+
**How it works:**
|
|
323
|
+
|
|
324
|
+
1. **Fetch** — Pulls issue details (title, body, labels) via `gh` or `glab` CLI
|
|
325
|
+
2. **Generate** — AI creates JSON test actions based on the issue description
|
|
326
|
+
3. **Run** — Optionally executes the tests immediately to verify if a bug is reproducible
|
|
256
327
|
|
|
257
328
|
```bash
|
|
258
|
-
|
|
259
|
-
|
|
329
|
+
# Fetch and display
|
|
330
|
+
e2e-runner issue https://github.com/owner/repo/issues/42
|
|
331
|
+
|
|
332
|
+
# Generate a test file via Claude API
|
|
333
|
+
e2e-runner issue https://github.com/owner/repo/issues/42 --generate
|
|
334
|
+
|
|
335
|
+
# Generate + run + report
|
|
336
|
+
e2e-runner issue https://github.com/owner/repo/issues/42 --verify
|
|
337
|
+
# -> "BUG CONFIRMED" or "NOT REPRODUCIBLE"
|
|
260
338
|
```
|
|
261
339
|
|
|
262
|
-
|
|
340
|
+
In Claude Code, just ask:
|
|
341
|
+
> "Fetch issue #42 and create E2E tests for it"
|
|
263
342
|
|
|
264
|
-
|
|
343
|
+
**Bug verification logic:** Generated tests assert the **correct** behavior. Test failure = bug confirmed. All tests pass = not reproducible.
|
|
265
344
|
|
|
266
|
-
|
|
267
|
-
jobs:
|
|
268
|
-
e2e:
|
|
269
|
-
runs-on: ubuntu-latest
|
|
270
|
-
steps:
|
|
271
|
-
- uses: actions/checkout@v4
|
|
272
|
-
- uses: actions/setup-node@v4
|
|
273
|
-
with:
|
|
274
|
-
node-version: 20
|
|
275
|
-
- run: npm ci
|
|
276
|
-
- run: npx e2e-runner pool start
|
|
277
|
-
- run: npx e2e-runner run --all --output junit
|
|
278
|
-
- uses: mikepenz/action-junit-report@v4
|
|
279
|
-
if: always()
|
|
280
|
-
with:
|
|
281
|
-
report_paths: e2e/screenshots/junit.xml
|
|
282
|
-
```
|
|
345
|
+
**Auth:** GitHub requires `gh` CLI, GitLab requires `glab` CLI. Self-hosted GitLab is supported.
|
|
283
346
|
|
|
284
|
-
|
|
347
|
+
---
|
|
285
348
|
|
|
286
|
-
|
|
287
|
-
|------|---------|
|
|
288
|
-
| `0` | All tests passed |
|
|
289
|
-
| `1` | One or more tests failed |
|
|
349
|
+
## Learning System
|
|
290
350
|
|
|
291
|
-
|
|
351
|
+
The runner learns from every test run — building knowledge about your test suite over time.
|
|
292
352
|
|
|
293
|
-
|
|
294
|
-
import { createRunner } from '@matware/e2e-runner';
|
|
353
|
+
Query insights via the `e2e_learnings` MCP tool:
|
|
295
354
|
|
|
296
|
-
|
|
355
|
+
| Query | Returns |
|
|
356
|
+
|-------|---------|
|
|
357
|
+
| `summary` | Full health overview: pass rate, flaky tests, unstable selectors, API issues |
|
|
358
|
+
| `flaky` | Tests that pass only after retries |
|
|
359
|
+
| `selectors` | CSS selectors with high failure rates |
|
|
360
|
+
| `pages` | Pages with console errors, network failures, load time issues |
|
|
361
|
+
| `apis` | API endpoints with error rates and latency (auto-normalized: UUIDs, hashes, IDs) |
|
|
362
|
+
| `errors` | Most frequent error patterns, categorized |
|
|
363
|
+
| `trends` | Pass rate over time (auto-switches to hourly when all data is from one day) |
|
|
364
|
+
| `test:<name>` | Drill-down history for a specific test |
|
|
365
|
+
| `page:<path>` | Drill-down history for a specific page |
|
|
366
|
+
| `selector:<value>` | Drill-down history for a specific selector |
|
|
297
367
|
|
|
298
|
-
|
|
299
|
-
|
|
368
|
+
**Storage & export:**
|
|
369
|
+
- SQLite (`~/.e2e-runner/dashboard.db`) — default, zero setup
|
|
370
|
+
- Neo4j knowledge graph — optional, for relationship-based analysis. Manage via `e2e_neo4j` MCP tool or `docker compose`
|
|
371
|
+
- Markdown report (`e2e/learnings.md`) — auto-generated after each run
|
|
300
372
|
|
|
301
|
-
|
|
302
|
-
const report = await runner.runSuite('auth');
|
|
373
|
+
**Test narration:** Each test run generates a human-readable narrative of what happened step by step, visible in the CLI output and the dashboard.
|
|
303
374
|
|
|
304
|
-
|
|
305
|
-
const report = await runner.runFile('e2e/tests/login.json');
|
|
375
|
+
---
|
|
306
376
|
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
],
|
|
315
|
-
},
|
|
316
|
-
]);
|
|
377
|
+
## Web Dashboard
|
|
378
|
+
|
|
379
|
+
Real-time UI for running tests, viewing results, screenshots, and network logs.
|
|
380
|
+
|
|
381
|
+
```bash
|
|
382
|
+
e2e-runner dashboard # Start on default port 8484
|
|
383
|
+
e2e-runner dashboard --port 9090 # Custom port
|
|
317
384
|
```
|
|
318
385
|
|
|
319
|
-
###
|
|
386
|
+
### Live Execution
|
|
320
387
|
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
388
|
+
Monitor tests in real-time with step-by-step progress, durations, and active worker count.
|
|
389
|
+
|
|
390
|
+
<p align="center">
|
|
391
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-live-running.png" alt="Dashboard - Live test execution" width="800" />
|
|
392
|
+
</p>
|
|
393
|
+
|
|
394
|
+
### Test Suites
|
|
395
|
+
|
|
396
|
+
Browse all test suites across multiple projects. Run a single suite or all tests with one click.
|
|
397
|
+
|
|
398
|
+
<p align="center">
|
|
399
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-suites.png" alt="Dashboard - Test suites grid" width="800" />
|
|
400
|
+
</p>
|
|
401
|
+
|
|
402
|
+
### Run History
|
|
403
|
+
|
|
404
|
+
Track pass rate trends with the built-in chart. Click any row to expand full detail with per-test results, screenshot hashes, and errors.
|
|
405
|
+
|
|
406
|
+
<p align="center">
|
|
407
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-runs.png" alt="Dashboard - Run history" width="800" />
|
|
408
|
+
</p>
|
|
409
|
+
|
|
410
|
+
### Run Detail
|
|
411
|
+
|
|
412
|
+
Expanded view with PASS/FAIL badges, screenshot thumbnails with copyable hashes (`ss:77c28b5a`), formatted console errors, and network request logs.
|
|
413
|
+
|
|
414
|
+
<p align="center">
|
|
415
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-run-detail.png" alt="Dashboard - Run detail" width="800" />
|
|
416
|
+
</p>
|
|
417
|
+
|
|
418
|
+
### Screenshot Gallery
|
|
419
|
+
|
|
420
|
+
Browse all captured screenshots with hash search. Includes action screenshots, error screenshots, and verification captures.
|
|
421
|
+
|
|
422
|
+
<p align="center">
|
|
423
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-screenshots-gallery.png" alt="Dashboard - Screenshot gallery" width="800" />
|
|
424
|
+
</p>
|
|
425
|
+
|
|
426
|
+
### Pool Status
|
|
427
|
+
|
|
428
|
+
Monitor Chrome pool health: available slots, running sessions, memory pressure.
|
|
429
|
+
|
|
430
|
+
<p align="center">
|
|
431
|
+
<img src="https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/docs/screenshots/blog-dashboard-pool-status.png" alt="Dashboard - Pool status" width="800" />
|
|
432
|
+
</p>
|
|
433
|
+
|
|
434
|
+
---
|
|
435
|
+
|
|
436
|
+
## Screenshot Capture
|
|
437
|
+
|
|
438
|
+
Capture screenshots of any URL on demand — no test suite required:
|
|
439
|
+
|
|
440
|
+
```bash
|
|
441
|
+
e2e-runner capture https://example.com
|
|
442
|
+
e2e-runner capture https://example.com --full-page --selector ".loaded" --delay 2000
|
|
329
443
|
```
|
|
330
444
|
|
|
331
|
-
|
|
445
|
+
Via MCP, the `e2e_capture` tool supports `authToken` and `authStorageKey` for authenticated pages — it injects the token into localStorage before navigating.
|
|
446
|
+
|
|
447
|
+
Every screenshot gets a deterministic hash (`ss:a3f2b1c9`). Use `e2e_screenshot` to retrieve any screenshot by hash — it returns the image with metadata (test name, step, type).
|
|
448
|
+
|
|
449
|
+
---
|
|
450
|
+
|
|
451
|
+
## Claude Code Integration
|
|
332
452
|
|
|
333
|
-
The package
|
|
453
|
+
The package ships as a **Claude Code plugin** — a single install that gives Claude native access to the test runner, teaches it the optimal workflow, and adds slash commands and specialized agents.
|
|
334
454
|
|
|
335
|
-
|
|
455
|
+
### Install as Plugin (recommended)
|
|
336
456
|
|
|
337
457
|
```bash
|
|
338
|
-
claude
|
|
339
|
-
-- npx -y -p @matware/e2e-runner e2e-runner-mcp
|
|
458
|
+
claude plugin install npm:@matware/e2e-runner
|
|
340
459
|
```
|
|
341
460
|
|
|
342
|
-
**
|
|
461
|
+
**What you get:**
|
|
462
|
+
|
|
463
|
+
| Component | Description |
|
|
464
|
+
|-----------|-------------|
|
|
465
|
+
| **13 MCP tools** | Run tests, create test files, capture screenshots, query network logs, manage dashboard, verify issues, query learnings |
|
|
466
|
+
| **Skill** | Teaches Claude the full e2e-runner workflow — how to combine tools, interpret results, debug failures, create tests |
|
|
467
|
+
| **3 Commands** | `/e2e-runner:run` — run & analyze tests<br>`/e2e-runner:create-test` — explore UI and create tests<br>`/e2e-runner:verify-issue <url>` — verify GitHub/GitLab bugs |
|
|
468
|
+
| **2 Agents** | **test-analyzer** — diagnoses failures, analyzes flaky tests, drills into network errors<br>**test-creator** — explores UI, discovers selectors, designs and validates tests |
|
|
469
|
+
|
|
470
|
+
### Install MCP-only (alternative)
|
|
471
|
+
|
|
472
|
+
If you only want the 13 MCP tools without skills, commands, or agents:
|
|
343
473
|
|
|
344
474
|
```bash
|
|
345
475
|
claude mcp add --transport stdio --scope user e2e-runner \
|
|
346
|
-
--
|
|
476
|
+
-- npx -y -p @matware/e2e-runner e2e-runner-mcp
|
|
347
477
|
```
|
|
348
478
|
|
|
479
|
+
### Slash Commands
|
|
480
|
+
|
|
481
|
+
| Command | Description |
|
|
482
|
+
|---------|-------------|
|
|
483
|
+
| `/e2e-runner:run` | Check pool, list suites, run tests, analyze results with screenshots and network drill-down |
|
|
484
|
+
| `/e2e-runner:create-test` | Explore the UI with screenshots, find selectors in source code, design test actions, create and validate |
|
|
485
|
+
| `/e2e-runner:verify-issue <url>` | Fetch a GitHub/GitLab issue, create tests that verify correct behavior, report bug confirmed or not reproducible |
|
|
486
|
+
|
|
349
487
|
### MCP Tools
|
|
350
488
|
|
|
351
489
|
| Tool | Description |
|
|
352
490
|
|------|-------------|
|
|
353
|
-
| `e2e_run` | Run tests
|
|
491
|
+
| `e2e_run` | Run tests: all suites, by name, or by file. Supports `concurrency`, `baseUrl`, `retries`, `failOnNetworkError` overrides. Returns verification results if tests have `expect`. |
|
|
354
492
|
| `e2e_list` | List available test suites with test names and counts |
|
|
355
|
-
| `e2e_create_test` | Create a new test JSON file |
|
|
356
|
-
| `
|
|
357
|
-
| `
|
|
358
|
-
| `
|
|
493
|
+
| `e2e_create_test` | Create a new test JSON file with name, tests, and optional hooks |
|
|
494
|
+
| `e2e_create_module` | Create a reusable module with parameterized actions |
|
|
495
|
+
| `e2e_pool_status` | Check Chrome pool availability, running sessions, capacity |
|
|
496
|
+
| `e2e_screenshot` | Retrieve a screenshot by hash (`ss:a3f2b1c9`). Returns image + metadata |
|
|
497
|
+
| `e2e_capture` | Capture screenshot of any URL. Supports `authToken`, `fullPage`, `selector`, `delay` |
|
|
498
|
+
| `e2e_dashboard_start` | Start the web dashboard |
|
|
499
|
+
| `e2e_dashboard_stop` | Stop the web dashboard |
|
|
500
|
+
| `e2e_issue` | Fetch GitHub/GitLab issue and generate tests. `mode: "prompt"` or `mode: "verify"` |
|
|
501
|
+
| `e2e_network_logs` | Query network request/response logs by `runDbId`. Filter by test name, method, status, URL pattern. Supports headers and bodies |
|
|
502
|
+
| `e2e_learnings` | Query the learning system: `summary`, `flaky`, `selectors`, `pages`, `apis`, `errors`, `trends` |
|
|
503
|
+
| `e2e_neo4j` | Manage Neo4j knowledge graph container: `start`, `stop`, `status` |
|
|
504
|
+
|
|
505
|
+
> **Note:** Pool start/stop are CLI-only (`e2e-runner pool start|stop`) — not exposed via MCP to prevent killing active sessions.
|
|
506
|
+
|
|
507
|
+
### What You Can Ask Claude Code
|
|
359
508
|
|
|
360
|
-
>
|
|
509
|
+
> "Run all E2E tests"
|
|
510
|
+
> "Create a test that verifies the checkout flow"
|
|
511
|
+
> "What tests are flaky? Show me the learning summary"
|
|
512
|
+
> "Capture a screenshot of /dashboard with auth"
|
|
513
|
+
> "Fetch issue #42 and create tests for it"
|
|
514
|
+
> "What's the API error rate for the last 7 days?"
|
|
361
515
|
|
|
362
|
-
|
|
516
|
+
---
|
|
363
517
|
|
|
364
|
-
|
|
518
|
+
## Network Error Handling
|
|
365
519
|
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
520
|
+
### Explicit Assertion
|
|
521
|
+
|
|
522
|
+
Place `assert_no_network_errors` after critical page loads:
|
|
523
|
+
|
|
524
|
+
```json
|
|
525
|
+
{ "type": "goto", "value": "/dashboard" },
|
|
526
|
+
{ "type": "wait", "selector": ".loaded" },
|
|
527
|
+
{ "type": "assert_no_network_errors" }
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
### Global Flag
|
|
369
531
|
|
|
370
|
-
|
|
532
|
+
Set `failOnNetworkError: true` to automatically fail any test with network errors:
|
|
371
533
|
|
|
372
534
|
```bash
|
|
373
|
-
|
|
374
|
-
# e2e-runner: ... - Connected
|
|
535
|
+
e2e-runner run --all --fail-on-network-error
|
|
375
536
|
```
|
|
376
537
|
|
|
377
|
-
|
|
538
|
+
When disabled (default), the runner still collects and reports network errors — the MCP response includes a warning when tests pass but have network errors.
|
|
539
|
+
|
|
540
|
+
### Full Network Logging
|
|
378
541
|
|
|
379
|
-
|
|
542
|
+
All XHR/fetch requests are captured with: URL, method, status, duration, request/response headers, and response body (truncated at 50KB). Viewable in the dashboard with expandable request detail rows.
|
|
380
543
|
|
|
381
|
-
|
|
544
|
+
**MCP drill-down flow:**
|
|
382
545
|
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
546
|
+
```
|
|
547
|
+
1. e2e_run → compact networkSummary + runDbId
|
|
548
|
+
2. e2e_network_logs(runDbId) → all requests (url, method, status, duration)
|
|
549
|
+
3. e2e_network_logs(runDbId, errorsOnly: true) → only failed requests
|
|
550
|
+
4. e2e_network_logs(runDbId, includeHeaders: true) → with headers
|
|
551
|
+
5. e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
|
|
552
|
+
```
|
|
386
553
|
|
|
387
|
-
|
|
554
|
+
The `e2e_run` response stays compact (~5KB) regardless of how many requests were captured. Use `e2e_network_logs` with the returned `runDbId` to drill into details on demand.
|
|
388
555
|
|
|
389
|
-
|
|
556
|
+
---
|
|
390
557
|
|
|
391
|
-
|
|
558
|
+
## Hooks
|
|
392
559
|
|
|
393
|
-
|
|
560
|
+
Run actions at lifecycle points. Define globally in config or per-suite:
|
|
561
|
+
|
|
562
|
+
```json
|
|
563
|
+
{
|
|
564
|
+
"hooks": {
|
|
565
|
+
"beforeAll": [{ "type": "goto", "value": "/setup" }],
|
|
566
|
+
"beforeEach": [{ "type": "goto", "value": "/" }],
|
|
567
|
+
"afterEach": [{ "type": "screenshot", "value": "after.png" }],
|
|
568
|
+
"afterAll": []
|
|
569
|
+
},
|
|
570
|
+
"tests": [...]
|
|
571
|
+
}
|
|
572
|
+
```
|
|
573
|
+
|
|
574
|
+
> **Important:** `beforeAll` runs on a separate browser page that is closed before tests start. Use `beforeEach` for state that tests need (cookies, localStorage, auth tokens).
|
|
575
|
+
|
|
576
|
+
---
|
|
577
|
+
|
|
578
|
+
## CLI
|
|
394
579
|
|
|
395
580
|
```bash
|
|
396
|
-
#
|
|
397
|
-
e2e-runner
|
|
581
|
+
# Run tests
|
|
582
|
+
e2e-runner run --all # All suites
|
|
583
|
+
e2e-runner run --suite auth # Single suite
|
|
584
|
+
e2e-runner run --tests path/to.json # Specific file
|
|
585
|
+
e2e-runner run --inline '<json>' # Inline JSON
|
|
398
586
|
|
|
399
|
-
#
|
|
400
|
-
e2e-runner
|
|
401
|
-
#
|
|
587
|
+
# Pool management (CLI only, not MCP)
|
|
588
|
+
e2e-runner pool start # Start Chrome container
|
|
589
|
+
e2e-runner pool stop # Stop Chrome container
|
|
590
|
+
e2e-runner pool status # Check pool health
|
|
402
591
|
|
|
403
|
-
#
|
|
404
|
-
e2e-runner issue
|
|
405
|
-
|
|
592
|
+
# Issue-to-test
|
|
593
|
+
e2e-runner issue <url> # Fetch issue
|
|
594
|
+
e2e-runner issue <url> --generate # Generate test via AI
|
|
595
|
+
e2e-runner issue <url> --verify # Generate + run + report
|
|
596
|
+
|
|
597
|
+
# Dashboard
|
|
598
|
+
e2e-runner dashboard # Start web dashboard
|
|
406
599
|
|
|
407
|
-
#
|
|
408
|
-
e2e-runner
|
|
600
|
+
# Other
|
|
601
|
+
e2e-runner list # List available suites
|
|
602
|
+
e2e-runner capture <url> # On-demand screenshot
|
|
603
|
+
e2e-runner init # Scaffold project
|
|
409
604
|
```
|
|
410
605
|
|
|
411
|
-
###
|
|
606
|
+
### CLI Options
|
|
412
607
|
|
|
413
|
-
|
|
608
|
+
| Flag | Default | Description |
|
|
609
|
+
|------|---------|-------------|
|
|
610
|
+
| `--base-url <url>` | `http://host.docker.internal:3000` | Application base URL |
|
|
611
|
+
| `--pool-url <ws>` | `ws://localhost:3333` | Chrome pool WebSocket URL |
|
|
612
|
+
| `--concurrency <n>` | `3` | Parallel test workers |
|
|
613
|
+
| `--retries <n>` | `0` | Retry failed tests N times |
|
|
614
|
+
| `--action-retries <n>` | `0` | Retry failed actions N times |
|
|
615
|
+
| `--test-timeout <ms>` | `60000` | Per-test timeout |
|
|
616
|
+
| `--timeout <ms>` | `10000` | Default action timeout |
|
|
617
|
+
| `--output <format>` | `json` | Report: `json`, `junit`, `both` |
|
|
618
|
+
| `--env <name>` | `default` | Environment profile |
|
|
619
|
+
| `--fail-on-network-error` | `false` | Fail tests with network errors |
|
|
620
|
+
| `--project-name <name>` | dir name | Project display name |
|
|
414
621
|
|
|
415
|
-
|
|
622
|
+
---
|
|
416
623
|
|
|
417
|
-
|
|
624
|
+
## Configuration
|
|
418
625
|
|
|
419
|
-
|
|
626
|
+
Create `e2e.config.js` in your project root:
|
|
420
627
|
|
|
421
|
-
|
|
422
|
-
|
|
628
|
+
```js
|
|
629
|
+
export default {
|
|
630
|
+
baseUrl: 'http://host.docker.internal:3000',
|
|
631
|
+
concurrency: 4,
|
|
632
|
+
retries: 2,
|
|
633
|
+
actionRetries: 1,
|
|
634
|
+
testTimeout: 30000,
|
|
635
|
+
outputFormat: 'both',
|
|
636
|
+
failOnNetworkError: true,
|
|
637
|
+
exclude: ['explore-*', 'debug-*'],
|
|
423
638
|
|
|
424
|
-
|
|
639
|
+
hooks: {
|
|
640
|
+
beforeEach: [{ type: 'goto', value: '/' }],
|
|
641
|
+
},
|
|
425
642
|
|
|
426
|
-
|
|
643
|
+
environments: {
|
|
644
|
+
staging: { baseUrl: 'https://staging.example.com' },
|
|
645
|
+
production: { baseUrl: 'https://example.com', concurrency: 5 },
|
|
646
|
+
},
|
|
647
|
+
};
|
|
648
|
+
```
|
|
427
649
|
|
|
428
|
-
|
|
650
|
+
### Config Priority (highest wins)
|
|
429
651
|
|
|
430
|
-
|
|
652
|
+
1. CLI flags
|
|
653
|
+
2. Environment variables
|
|
654
|
+
3. Config file (`e2e.config.js` or `e2e.config.json`)
|
|
655
|
+
4. Defaults
|
|
431
656
|
|
|
432
|
-
|
|
657
|
+
When `--env <name>` is set, the matching profile overrides everything.
|
|
433
658
|
|
|
434
|
-
|
|
435
|
-
e2e-runner dashboard # Start on default port 8484
|
|
436
|
-
e2e-runner dashboard --port 9090 # Custom port
|
|
437
|
-
```
|
|
659
|
+
---
|
|
438
660
|
|
|
439
|
-
|
|
661
|
+
## CI/CD
|
|
440
662
|
|
|
441
|
-
|
|
663
|
+
### JUnit XML
|
|
442
664
|
|
|
665
|
+
```bash
|
|
666
|
+
e2e-runner run --all --output junit
|
|
443
667
|
```
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
668
|
+
|
|
669
|
+
### GitHub Actions
|
|
670
|
+
|
|
671
|
+
```yaml
|
|
672
|
+
jobs:
|
|
673
|
+
e2e:
|
|
674
|
+
runs-on: ubuntu-latest
|
|
675
|
+
steps:
|
|
676
|
+
- uses: actions/checkout@v4
|
|
677
|
+
- uses: actions/setup-node@v4
|
|
678
|
+
with:
|
|
679
|
+
node-version: 20
|
|
680
|
+
- run: npm ci
|
|
681
|
+
- run: npx e2e-runner pool start
|
|
682
|
+
- run: npx e2e-runner run --all --output junit
|
|
683
|
+
- uses: mikepenz/action-junit-report@v4
|
|
684
|
+
if: always()
|
|
685
|
+
with:
|
|
686
|
+
report_paths: e2e/screenshots/junit.xml
|
|
461
687
|
```
|
|
462
688
|
|
|
463
|
-
|
|
689
|
+
---
|
|
690
|
+
|
|
691
|
+
## Programmatic API
|
|
692
|
+
|
|
693
|
+
```js
|
|
694
|
+
import { createRunner } from '@matware/e2e-runner';
|
|
695
|
+
|
|
696
|
+
const runner = await createRunner({ baseUrl: 'http://localhost:3000' });
|
|
464
697
|
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
698
|
+
const report = await runner.runAll();
|
|
699
|
+
const report = await runner.runSuite('auth');
|
|
700
|
+
const report = await runner.runFile('e2e/tests/login.json');
|
|
701
|
+
const report = await runner.runTests([
|
|
702
|
+
{ name: 'quick-check', actions: [{ type: 'goto', value: '/' }] },
|
|
703
|
+
]);
|
|
704
|
+
```
|
|
469
705
|
|
|
470
|
-
|
|
706
|
+
---
|
|
471
707
|
|
|
472
708
|
## Requirements
|
|
473
709
|
|
|
@@ -478,14 +714,4 @@ The `baseUrl` defaults to `http://host.docker.internal:3000` because Chrome runs
|
|
|
478
714
|
|
|
479
715
|
Copyright 2025 Matias Aguirre (fastslack)
|
|
480
716
|
|
|
481
|
-
Licensed under the Apache License, Version 2.0 (
|
|
482
|
-
you may not use this file except in compliance with the License.
|
|
483
|
-
You may obtain a copy of the License at
|
|
484
|
-
|
|
485
|
-
http://www.apache.org/licenses/LICENSE-2.0
|
|
486
|
-
|
|
487
|
-
Unless required by applicable law or agreed to in writing, software
|
|
488
|
-
distributed under the License is distributed on an "AS IS" BASIS,
|
|
489
|
-
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
490
|
-
See the License for the specific language governing permissions and
|
|
491
|
-
limitations under the License.
|
|
717
|
+
Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.
|