@matware/e2e-runner 1.2.1 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +21 -0
- package/.mcp.json +2 -2
- package/.opencode/commands/create-test.md +63 -0
- package/.opencode/commands/run.md +50 -0
- package/.opencode/commands/verify-issue.md +62 -0
- package/.opencode/skills/e2e-testing/SKILL.md +181 -0
- package/.opencode/skills/e2e-testing/references/action-types.md +143 -0
- package/.opencode/skills/e2e-testing/references/auth-strategies.md +91 -0
- package/.opencode/skills/e2e-testing/references/graphql.md +59 -0
- package/.opencode/skills/e2e-testing/references/issue-verification.md +59 -0
- package/.opencode/skills/e2e-testing/references/multi-pool.md +60 -0
- package/.opencode/skills/e2e-testing/references/network-debugging.md +62 -0
- package/.opencode/skills/e2e-testing/references/test-json-format.md +163 -0
- package/.opencode/skills/e2e-testing/references/troubleshooting.md +224 -0
- package/.opencode/skills/e2e-testing/references/variables.md +41 -0
- package/.opencode/skills/e2e-testing/references/visual-verification.md +89 -0
- package/OPENCODE.md +166 -0
- package/README.md +581 -55
- package/agents/test-creator.md +54 -1
- package/agents/test-improver.md +37 -0
- package/bin/cli.js +408 -16
- package/commands/create-test.md +16 -1
- package/opencode.json +11 -0
- package/package.json +7 -2
- package/scripts/setup-opencode.sh +113 -0
- package/skills/e2e-testing/SKILL.md +10 -3
- package/skills/e2e-testing/references/action-types.md +48 -5
- package/skills/e2e-testing/references/auth-strategies.md +91 -0
- package/skills/e2e-testing/references/graphql.md +59 -0
- package/skills/e2e-testing/references/issue-verification.md +59 -0
- package/skills/e2e-testing/references/multi-pool.md +60 -0
- package/skills/e2e-testing/references/network-debugging.md +62 -0
- package/skills/e2e-testing/references/test-json-format.md +4 -0
- package/skills/e2e-testing/references/troubleshooting.md +44 -2
- package/skills/e2e-testing/references/variables.md +41 -0
- package/skills/e2e-testing/references/visual-verification.md +89 -0
- package/src/actions.js +324 -2
- package/src/ai-generate.js +58 -8
- package/src/config.js +143 -0
- package/src/dashboard.js +145 -13
- package/src/db.js +130 -2
- package/src/index.js +7 -6
- package/src/learner-sqlite.js +304 -0
- package/src/learner.js +8 -3
- package/src/mcp-tools.js +1121 -43
- package/src/module-resolver.js +37 -0
- package/src/narrate.js +37 -0
- package/src/pool-manager.js +223 -0
- package/src/reporter.js +82 -1
- package/src/runner.js +157 -28
- package/src/sync/auth.js +354 -0
- package/src/sync/client.js +572 -0
- package/src/sync/hub-routes.js +816 -0
- package/src/sync/index.js +68 -0
- package/src/sync/middleware.js +347 -0
- package/src/sync/queue.js +209 -0
- package/src/sync/schema.js +540 -0
- package/src/verify.js +10 -7
- package/src/watch.js +384 -0
- package/templates/build-dashboard.js +47 -6
- package/templates/dashboard/js/api.js +60 -0
- package/templates/dashboard/js/init.js +13 -0
- package/templates/dashboard/js/keyboard.js +46 -0
- package/templates/dashboard/js/state.js +40 -0
- package/templates/dashboard/js/toast.js +41 -0
- package/templates/dashboard/js/utils.js +196 -0
- package/templates/dashboard/js/view-live.js +143 -0
- package/templates/dashboard/js/view-runs.js +572 -0
- package/templates/dashboard/js/view-tests.js +294 -0
- package/templates/dashboard/js/view-watch.js +242 -0
- package/templates/dashboard/js/websocket.js +110 -0
- package/templates/dashboard/styles/base.css +69 -0
- package/templates/dashboard/styles/components.css +110 -0
- package/templates/dashboard/styles/view-live.css +74 -0
- package/templates/dashboard/styles/view-runs.css +207 -0
- package/templates/dashboard/styles/view-tests.css +96 -0
- package/templates/dashboard/styles/view-watch.css +53 -0
- package/templates/dashboard/template.html +165 -99
- package/templates/dashboard.html +1596 -541
- package/templates/sample-test.json +0 -8
- package/templates/dashboard/app.js +0 -1152
- package/templates/dashboard/styles.css +0 -413
package/README.md
CHANGED
|
@@ -14,6 +14,7 @@
|
|
|
14
14
|
<img src="https://img.shields.io/npm/l/@matware/e2e-runner" alt="license" />
|
|
15
15
|
<img src="https://img.shields.io/badge/MCP-compatible-green" alt="MCP compatible" />
|
|
16
16
|
<img src="https://img.shields.io/badge/AI--native-Claude%20Code-blueviolet" alt="AI native" />
|
|
17
|
+
<img src="https://img.shields.io/badge/AI--native-OpenCode-orange" alt="OpenCode compatible" />
|
|
17
18
|
</p>
|
|
18
19
|
|
|
19
20
|
<p align="center">
|
|
@@ -26,32 +27,6 @@
|
|
|
26
27
|
|
|
27
28
|
But what makes it truly different is its **deep AI integration**. With a built-in [MCP server](https://modelcontextprotocol.io/), Claude Code can create tests from a conversation, run them, read the results, capture screenshots, and even visually verify that pages look correct — all without leaving the chat. Paste a GitHub issue URL and get a runnable test back. That's the workflow.
|
|
28
29
|
|
|
29
|
-
### What you get
|
|
30
|
-
|
|
31
|
-
🧪 **Zero-code tests** — JSON files that anyone on your team can read and write. No JavaScript, no compilation, no framework lock-in.
|
|
32
|
-
|
|
33
|
-
🤖 **AI-powered testing** — Claude Code creates, executes, and debugs tests natively through 13 MCP tools. Ask it to "test the checkout flow" and it builds the JSON, runs it, and reports back.
|
|
34
|
-
|
|
35
|
-
🐛 **Issue-to-Test pipeline** — Paste a GitHub or GitLab issue URL. The runner fetches it, generates E2E tests, runs them, and tells you: *bug confirmed* or *not reproducible*.
|
|
36
|
-
|
|
37
|
-
👁️ **Visual verification** — Describe what the page should look like in plain English. The AI captures a screenshot and judges pass/fail against your description. No pixel-diffing setup needed.
|
|
38
|
-
|
|
39
|
-
🧠 **Learning system** — Tracks test stability across runs. Detects flaky tests, unstable selectors, slow APIs, and error patterns — then surfaces actionable insights.
|
|
40
|
-
|
|
41
|
-
⚡ **Parallel execution** — Run N tests simultaneously against a shared Chrome pool (browserless/chrome). Serial mode available for tests that share state.
|
|
42
|
-
|
|
43
|
-
📊 **Real-time dashboard** — Live execution view, run history with pass-rate charts, screenshot gallery with hash-based search, expandable network request logs.
|
|
44
|
-
|
|
45
|
-
🔁 **Smart retries** — Test-level and action-level retries with configurable delays. Flaky tests are detected and flagged automatically.
|
|
46
|
-
|
|
47
|
-
📦 **Reusable modules** — Extract common flows (login, navigation, setup) into parameterized modules and reference them with `$use`.
|
|
48
|
-
|
|
49
|
-
🏗️ **CI-ready** — JUnit XML output, exit code 1 on failure, auto-captured error screenshots. Drop-in GitHub Actions example included.
|
|
50
|
-
|
|
51
|
-
🌐 **Multi-project** — One dashboard aggregates test results from all your projects. One Chrome pool serves them all.
|
|
52
|
-
|
|
53
|
-
🐳 **Portable** — Chrome runs in Docker, tests are JSON files in your repo. Works on any machine with Node.js and Docker.
|
|
54
|
-
|
|
55
30
|
### This is a test
|
|
56
31
|
|
|
57
32
|
```json
|
|
@@ -74,45 +49,232 @@ No imports. No `describe`/`it`. No compilation step. Just a JSON file that descr
|
|
|
74
49
|
|
|
75
50
|
---
|
|
76
51
|
|
|
77
|
-
##
|
|
52
|
+
## Getting Started
|
|
53
|
+
|
|
54
|
+
### Prerequisites
|
|
55
|
+
|
|
56
|
+
- **Node.js** >= 20
|
|
57
|
+
- **Docker** running (for the Chrome pool)
|
|
58
|
+
- Your app running on a known port (e.g. `http://localhost:3000`)
|
|
59
|
+
|
|
60
|
+
> **Why `host.docker.internal`?**
|
|
61
|
+
>
|
|
62
|
+
> Chrome runs inside a Docker container. From inside the container, `localhost` refers to the container itself — not your machine. The special hostname `host.docker.internal` resolves to your host machine, so Chrome can reach your locally running app.
|
|
63
|
+
>
|
|
64
|
+
> The default `baseUrl` is `http://host.docker.internal:3000`. If your app runs on a different port, change it in `e2e.config.js` after init.
|
|
65
|
+
>
|
|
66
|
+
> **Linux note:** On Docker Engine (not Docker Desktop), you may need to add `--add-host=host.docker.internal:host-gateway` to the Docker run flags, or use your machine's LAN IP directly as the `baseUrl`.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
### Path A: With Claude Code
|
|
71
|
+
|
|
72
|
+
If you use [Claude Code](https://docs.anthropic.com/en/docs/claude-code), this is the fastest path — Claude handles test creation and debugging for you.
|
|
78
73
|
|
|
79
|
-
**
|
|
74
|
+
**1. Install the package**
|
|
80
75
|
|
|
81
76
|
```bash
|
|
82
|
-
|
|
77
|
+
npm install --save-dev @matware/e2e-runner
|
|
83
78
|
```
|
|
84
79
|
|
|
85
|
-
**
|
|
80
|
+
**2. Scaffold the project structure**
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
npx e2e-runner init
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
This creates `e2e/tests/` with a sample test and `e2e/screenshots/` for captures.
|
|
87
|
+
|
|
88
|
+
**3. Configure your base URL**
|
|
89
|
+
|
|
90
|
+
Edit `e2e.config.js` and set `baseUrl` to match your app's port:
|
|
91
|
+
|
|
92
|
+
```js
|
|
93
|
+
export default {
|
|
94
|
+
baseUrl: 'http://host.docker.internal:3000', // change 3000 to your port
|
|
95
|
+
};
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
**4. Start the Chrome pool**
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
npx e2e-runner pool start
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
You should see:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
✓ Chrome pool started on port 3333 (max 3 sessions)
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
**5. Install the Claude Code plugin**
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
# Add the marketplace (one-time)
|
|
114
|
+
claude plugin marketplace add fastslack/mtw-e2e-runner
|
|
115
|
+
|
|
116
|
+
# Install the plugin
|
|
117
|
+
claude plugin install e2e-runner@matware
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The plugin gives Claude 13 MCP tools, a workflow skill, 3 slash commands, and 3 specialized agents.
|
|
121
|
+
|
|
122
|
+
**6. Ask Claude to run the sample test**
|
|
123
|
+
|
|
124
|
+
In Claude Code, just say:
|
|
125
|
+
|
|
126
|
+
> "Run all E2E tests"
|
|
127
|
+
|
|
128
|
+
Claude will check the pool, run the sample test, and report back:
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
==================================================
|
|
132
|
+
E2E RESULTS
|
|
133
|
+
==================================================
|
|
134
|
+
Total: 1
|
|
135
|
+
Passed: 1
|
|
136
|
+
Failed: 0
|
|
137
|
+
Rate: 100.00%
|
|
138
|
+
Duration: 1.23s
|
|
139
|
+
==================================================
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
From here, you can ask Claude to create new tests ("test the login flow"), debug failures, or verify GitHub issues.
|
|
143
|
+
|
|
144
|
+
---
|
|
145
|
+
|
|
146
|
+
### Path B: CLI Only
|
|
147
|
+
|
|
148
|
+
No AI required — use the runner directly from your terminal.
|
|
149
|
+
|
|
150
|
+
**1. Install the package**
|
|
86
151
|
|
|
87
152
|
```bash
|
|
88
|
-
# 1. Install
|
|
89
153
|
npm install --save-dev @matware/e2e-runner
|
|
154
|
+
```
|
|
90
155
|
|
|
91
|
-
|
|
156
|
+
**2. Scaffold the project structure**
|
|
157
|
+
|
|
158
|
+
```bash
|
|
92
159
|
npx e2e-runner init
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
This creates `e2e/tests/` with a sample test and `e2e/screenshots/` for captures.
|
|
163
|
+
|
|
164
|
+
**3. Configure your base URL**
|
|
93
165
|
|
|
94
|
-
|
|
166
|
+
Edit `e2e.config.js` and set `baseUrl` to match your app's port:
|
|
167
|
+
|
|
168
|
+
```js
|
|
169
|
+
export default {
|
|
170
|
+
baseUrl: 'http://host.docker.internal:3000', // change 3000 to your port
|
|
171
|
+
};
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
**4. Start the Chrome pool**
|
|
175
|
+
|
|
176
|
+
```bash
|
|
95
177
|
npx e2e-runner pool start
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
You should see:
|
|
96
181
|
|
|
97
|
-
|
|
182
|
+
```
|
|
183
|
+
✓ Chrome pool started on port 3333 (max 3 sessions)
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
**5. Run the sample test**
|
|
187
|
+
|
|
188
|
+
```bash
|
|
98
189
|
npx e2e-runner run --all
|
|
190
|
+
```
|
|
99
191
|
|
|
100
|
-
|
|
101
|
-
|
|
192
|
+
Expected output:
|
|
193
|
+
|
|
194
|
+
```
|
|
195
|
+
==================================================
|
|
196
|
+
E2E RESULTS
|
|
197
|
+
==================================================
|
|
198
|
+
Total: 1
|
|
199
|
+
Passed: 1
|
|
200
|
+
Failed: 0
|
|
201
|
+
Rate: 100.00%
|
|
202
|
+
Duration: 1.23s
|
|
203
|
+
==================================================
|
|
102
204
|
```
|
|
103
205
|
|
|
104
|
-
|
|
206
|
+
A screenshot is saved at `e2e/screenshots/homepage.png`.
|
|
207
|
+
|
|
208
|
+
**6. Write your first real test**
|
|
209
|
+
|
|
210
|
+
Create `e2e/tests/my-first-test.json`:
|
|
211
|
+
|
|
212
|
+
```json
|
|
213
|
+
[
|
|
214
|
+
{
|
|
215
|
+
"name": "homepage-visible",
|
|
216
|
+
"actions": [
|
|
217
|
+
{ "type": "goto", "value": "/" },
|
|
218
|
+
{ "type": "assert_visible", "selector": "body" },
|
|
219
|
+
{ "type": "screenshot", "value": "my-first-test.png" }
|
|
220
|
+
]
|
|
221
|
+
}
|
|
222
|
+
]
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
Run it:
|
|
105
226
|
|
|
106
227
|
```bash
|
|
107
|
-
|
|
108
|
-
|
|
228
|
+
npx e2e-runner run --suite my-first-test
|
|
229
|
+
```
|
|
109
230
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
231
|
+
---
|
|
232
|
+
|
|
233
|
+
### One-liner quickstart
|
|
234
|
+
|
|
235
|
+
If you want to skip the step-by-step and get everything running in one command:
|
|
236
|
+
|
|
237
|
+
```bash
|
|
238
|
+
curl -fsSL https://raw.githubusercontent.com/fastslack/mtw-e2e-runner/main/scripts/quickstart.sh | bash
|
|
113
239
|
```
|
|
114
240
|
|
|
115
|
-
|
|
241
|
+
> This installs the package, scaffolds the project, and starts the Chrome pool. You'll still need to configure your `baseUrl` afterwards.
|
|
242
|
+
|
|
243
|
+
### What's next?
|
|
244
|
+
|
|
245
|
+
- [Test Format](#test-format) — learn the full action vocabulary
|
|
246
|
+
- [Claude Code Integration](#claude-code-integration) — set up AI-powered testing
|
|
247
|
+
- [Visual Verification](#visual-verification) — describe expected pages in plain English
|
|
248
|
+
- [Issue-to-Test](#issue-to-test) — turn bug reports into executable tests
|
|
249
|
+
- [Web Dashboard](#web-dashboard) — monitor tests in real time
|
|
250
|
+
|
|
251
|
+
---
|
|
252
|
+
|
|
253
|
+
## What you get
|
|
254
|
+
|
|
255
|
+
🧪 **Zero-code tests** — JSON files that anyone on your team can read and write. No JavaScript, no compilation, no framework lock-in.
|
|
256
|
+
|
|
257
|
+
🤖 **AI-powered testing** — Claude Code creates, executes, and debugs tests natively through 13 MCP tools. Ask it to "test the checkout flow" and it builds the JSON, runs it, and reports back.
|
|
258
|
+
|
|
259
|
+
🐛 **Issue-to-Test pipeline** — Paste a GitHub or GitLab issue URL. The runner fetches it, generates E2E tests, runs them, and tells you: *bug confirmed* or *not reproducible*.
|
|
260
|
+
|
|
261
|
+
👁️ **Visual verification** — Describe what the page should look like in plain English. The AI captures a screenshot and judges pass/fail against your description. No pixel-diffing setup needed.
|
|
262
|
+
|
|
263
|
+
🧠 **Learning system** — Tracks test stability across runs. Detects flaky tests, unstable selectors, slow APIs, and error patterns — then surfaces actionable insights.
|
|
264
|
+
|
|
265
|
+
⚡ **Parallel execution** — Run N tests simultaneously against a shared Chrome pool (browserless/chrome). Serial mode available for tests that share state.
|
|
266
|
+
|
|
267
|
+
📊 **Real-time dashboard** — Live execution view, run history with pass-rate charts, screenshot gallery with hash-based search, expandable network request logs.
|
|
268
|
+
|
|
269
|
+
🔁 **Smart retries** — Test-level and action-level retries with configurable delays. Flaky tests are detected and flagged automatically.
|
|
270
|
+
|
|
271
|
+
📦 **Reusable modules** — Extract common flows (login, navigation, setup) into parameterized modules and reference them with `$use`.
|
|
272
|
+
|
|
273
|
+
🏗️ **CI-ready** — JUnit XML output, exit code 1 on failure, auto-captured error screenshots. Drop-in GitHub Actions example included.
|
|
274
|
+
|
|
275
|
+
🌐 **Multi-project** — One dashboard aggregates test results from all your projects. One Chrome pool serves them all.
|
|
276
|
+
|
|
277
|
+
🐳 **Portable** — Chrome runs in Docker, tests are JSON files in your repo. Works on any machine with Node.js and Docker.
|
|
116
278
|
|
|
117
279
|
---
|
|
118
280
|
|
|
@@ -126,8 +288,7 @@ Each `.json` file in `e2e/tests/` contains an array of tests. Each test has a `n
|
|
|
126
288
|
"name": "homepage-loads",
|
|
127
289
|
"actions": [
|
|
128
290
|
{ "type": "goto", "value": "/" },
|
|
129
|
-
{ "type": "
|
|
130
|
-
{ "type": "assert_text", "text": "Welcome" },
|
|
291
|
+
{ "type": "assert_visible", "selector": "body" },
|
|
131
292
|
{ "type": "assert_url", "value": "/" },
|
|
132
293
|
{ "type": "screenshot", "value": "homepage.png" }
|
|
133
294
|
]
|
|
@@ -245,22 +406,320 @@ Serial tests run one at a time **after** all parallel tests finish — preventin
|
|
|
245
406
|
|
|
246
407
|
---
|
|
247
408
|
|
|
409
|
+
## Testing Authenticated Apps
|
|
410
|
+
|
|
411
|
+
Most real-world apps require login before tests can interact with protected pages. E2E Runner provides multiple strategies — choose the one that matches your app's auth mechanism.
|
|
412
|
+
|
|
413
|
+
### Strategy 1: UI Login Flow (any app)
|
|
414
|
+
|
|
415
|
+
The most universal approach — fill in the login form like a real user. Works with **any** authentication system (session cookies, JWT, OAuth redirect, etc.):
|
|
416
|
+
|
|
417
|
+
```json
|
|
418
|
+
{
|
|
419
|
+
"hooks": {
|
|
420
|
+
"beforeEach": [
|
|
421
|
+
{ "type": "goto", "value": "/login" },
|
|
422
|
+
{ "type": "type", "selector": "#email", "value": "test@example.com" },
|
|
423
|
+
{ "type": "type", "selector": "#password", "value": "test-password" },
|
|
424
|
+
{ "type": "click", "text": "Sign In" },
|
|
425
|
+
{ "type": "wait", "selector": ".dashboard" }
|
|
426
|
+
]
|
|
427
|
+
},
|
|
428
|
+
"tests": [
|
|
429
|
+
{
|
|
430
|
+
"name": "profile-page",
|
|
431
|
+
"actions": [
|
|
432
|
+
{ "type": "goto", "value": "/profile" },
|
|
433
|
+
{ "type": "assert_text", "text": "My Profile" }
|
|
434
|
+
]
|
|
435
|
+
}
|
|
436
|
+
]
|
|
437
|
+
}
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
> **When to use:** You don't know or care how auth works internally. The browser handles cookies/tokens automatically after login — just like a real user.
|
|
441
|
+
|
|
442
|
+
### Strategy 2: JWT Token Injection (SPAs)
|
|
443
|
+
|
|
444
|
+
For single-page apps that store JWT tokens in `localStorage` or `sessionStorage`. Skip the login form entirely by injecting the token directly:
|
|
445
|
+
|
|
446
|
+
```json
|
|
447
|
+
{
|
|
448
|
+
"hooks": {
|
|
449
|
+
"beforeEach": [
|
|
450
|
+
{ "type": "goto", "value": "/" },
|
|
451
|
+
{ "type": "set_storage", "value": "accessToken=eyJhbGciOiJIUzI1NiIs..." },
|
|
452
|
+
{ "type": "goto", "value": "/dashboard" },
|
|
453
|
+
{ "type": "wait", "selector": ".dashboard-loaded" }
|
|
454
|
+
]
|
|
455
|
+
},
|
|
456
|
+
"tests": [...]
|
|
457
|
+
}
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
**Common storage key names** (depends on your app):
|
|
461
|
+
|
|
462
|
+
| Framework / Library | Typical key | Storage |
|
|
463
|
+
|---------------------|-------------|---------|
|
|
464
|
+
| Custom JWT | `accessToken`, `token`, `jwt` | localStorage |
|
|
465
|
+
| Auth0 SPA SDK | `@@auth0spajs@@::*` | localStorage |
|
|
466
|
+
| Firebase Auth | `firebase:authUser:*` | localStorage |
|
|
467
|
+
| AWS Amplify | `CognitoIdentityServiceProvider.*` | localStorage |
|
|
468
|
+
| Supabase | `sb-<ref>-auth-token` | localStorage |
|
|
469
|
+
| NextAuth (client) | `next-auth.session-token` | cookie (see Strategy 4) |
|
|
470
|
+
|
|
471
|
+
**Using `sessionStorage` instead:**
|
|
472
|
+
|
|
473
|
+
```json
|
|
474
|
+
{ "type": "set_storage", "value": "token=eyJhbG...", "selector": "session" }
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
**Asserting the token was stored correctly:**
|
|
478
|
+
|
|
479
|
+
```json
|
|
480
|
+
{ "type": "assert_storage", "value": "accessToken" }
|
|
481
|
+
{ "type": "assert_storage", "value": "accessToken=eyJhbG..." }
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
> **When to use:** Your SPA reads auth tokens from browser storage. Fastest strategy — no network round-trip for login.
|
|
485
|
+
|
|
486
|
+
### Strategy 3: Config-Level Auth Token
|
|
487
|
+
|
|
488
|
+
For apps where every test needs the same JWT token. Set it once in config — it's injected into `localStorage` before every `e2e_capture` and `e2e_issue --verify` run:
|
|
489
|
+
|
|
490
|
+
```js
|
|
491
|
+
// e2e.config.js
|
|
492
|
+
export default {
|
|
493
|
+
authToken: 'eyJhbGciOiJIUzI1NiIs...',
|
|
494
|
+
authStorageKey: 'accessToken', // default
|
|
495
|
+
};
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
Or via environment variables:
|
|
499
|
+
|
|
500
|
+
```bash
|
|
501
|
+
AUTH_TOKEN="eyJhbGciOiJIUzI1NiIs..." npx e2e-runner run --all
|
|
502
|
+
```
|
|
503
|
+
|
|
504
|
+
Or via CLI:
|
|
505
|
+
|
|
506
|
+
```bash
|
|
507
|
+
npx e2e-runner run --all --auth-token "eyJhbG..." --auth-storage-key "jwt"
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
MCP tools (`e2e_capture`, `e2e_issue`) also accept `authToken` and `authStorageKey` per call.
|
|
511
|
+
|
|
512
|
+
> **When to use:** All tests share the same user session and your app uses JWT in localStorage.
|
|
513
|
+
|
|
514
|
+
### Strategy 4: Cookie-Based Auth (server-rendered apps)
|
|
515
|
+
|
|
516
|
+
For apps that use HTTP cookies (Rails, Django, Laravel, Express sessions, NextAuth, etc.). Use `evaluate` to set cookies before navigating:
|
|
517
|
+
|
|
518
|
+
```json
|
|
519
|
+
{
|
|
520
|
+
"hooks": {
|
|
521
|
+
"beforeEach": [
|
|
522
|
+
{ "type": "goto", "value": "/" },
|
|
523
|
+
{ "type": "evaluate", "value": "document.cookie = 'session_id=abc123; path=/; SameSite=Lax'" },
|
|
524
|
+
{ "type": "goto", "value": "/dashboard" }
|
|
525
|
+
]
|
|
526
|
+
},
|
|
527
|
+
"tests": [...]
|
|
528
|
+
}
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
**Multiple cookies:**
|
|
532
|
+
|
|
533
|
+
```json
|
|
534
|
+
{ "type": "evaluate", "value": "document.cookie = 'session_id=abc123; path=/'; document.cookie = '_csrf_token=xyz789; path=/'" }
|
|
535
|
+
```
|
|
536
|
+
|
|
537
|
+
**For `HttpOnly` cookies** (can't be set via JavaScript), use the UI login strategy instead — the browser will store them automatically.
|
|
538
|
+
|
|
539
|
+
> **When to use:** Traditional server-rendered apps, or any app that authenticates via cookies.
|
|
540
|
+
|
|
541
|
+
### Strategy 5: HTTP Header Auth (API tests)
|
|
542
|
+
|
|
543
|
+
For API testing where you need to send `Authorization` headers with every request. Use `evaluate` to override `fetch`/`XMLHttpRequest`:
|
|
544
|
+
|
|
545
|
+
```json
|
|
546
|
+
{
|
|
547
|
+
"hooks": {
|
|
548
|
+
"beforeEach": [
|
|
549
|
+
{ "type": "goto", "value": "/" },
|
|
550
|
+
{ "type": "evaluate", "value": "const origFetch = window.fetch; window.fetch = (url, opts = {}) => { opts.headers = { ...opts.headers, 'Authorization': 'Bearer eyJhbG...' }; return origFetch(url, opts); }" }
|
|
551
|
+
]
|
|
552
|
+
},
|
|
553
|
+
"tests": [
|
|
554
|
+
{
|
|
555
|
+
"name": "api-returns-user",
|
|
556
|
+
"actions": [
|
|
557
|
+
{ "type": "evaluate", "value": "const res = await fetch('/api/me'); const data = await res.json(); if (data.email !== 'test@example.com') throw new Error('Wrong user: ' + data.email)" }
|
|
558
|
+
]
|
|
559
|
+
}
|
|
560
|
+
]
|
|
561
|
+
}
|
|
562
|
+
```
|
|
563
|
+
|
|
564
|
+
> **When to use:** API-level tests (with `--test-type api`) that need auth headers.
|
|
565
|
+
|
|
566
|
+
### Strategy 6: OAuth / SSO (external provider)
|
|
567
|
+
|
|
568
|
+
OAuth flows redirect to external providers (Google, GitHub, Okta, etc.) which can't be automated reliably. Common workarounds:
|
|
569
|
+
|
|
570
|
+
**Option A — Test environment bypass:** Most apps have a direct login endpoint for testing that skips OAuth:
|
|
571
|
+
|
|
572
|
+
```json
|
|
573
|
+
{ "type": "goto", "value": "/auth/test-login?user=test@example.com" }
|
|
574
|
+
```
|
|
575
|
+
|
|
576
|
+
**Option B — Pre-authenticated token:** Get a token from your auth provider's API and inject it:
|
|
577
|
+
|
|
578
|
+
```json
|
|
579
|
+
{
|
|
580
|
+
"hooks": {
|
|
581
|
+
"beforeEach": [
|
|
582
|
+
{ "type": "goto", "value": "/" },
|
|
583
|
+
{ "type": "set_storage", "value": "oidc.user:https://auth.example.com:client_id={\"access_token\":\"...\"}" }
|
|
584
|
+
]
|
|
585
|
+
}
|
|
586
|
+
}
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
**Option C — Session cookie from CI:** If your CI can authenticate via API, pass the session cookie as an env var:
|
|
590
|
+
|
|
591
|
+
```bash
|
|
592
|
+
SESSION=$(curl -s -c - https://api.example.com/auth/login -d '{"email":"test@example.com","password":"secret"}' | grep session_id | awk '{print $NF}')
|
|
593
|
+
AUTH_TOKEN="$SESSION" AUTH_STORAGE_KEY="session_id" npx e2e-runner run --all
|
|
594
|
+
```
|
|
595
|
+
|
|
596
|
+
> **When to use:** Apps with Google/GitHub/Okta/Auth0 login. You almost always need a test-environment backdoor.
|
|
597
|
+
|
|
598
|
+
### Reusable Auth Modules
|
|
599
|
+
|
|
600
|
+
Extract your auth strategy into a module so every test can reference it without duplication:
|
|
601
|
+
|
|
602
|
+
```json
|
|
603
|
+
// e2e/modules/login.json — UI login (universal)
|
|
604
|
+
{
|
|
605
|
+
"$module": "login",
|
|
606
|
+
"description": "Log in via the UI login form",
|
|
607
|
+
"params": {
|
|
608
|
+
"email": { "required": true, "description": "User email" },
|
|
609
|
+
"password": { "required": true, "description": "User password" },
|
|
610
|
+
"redirectTo": { "default": "/dashboard", "description": "Page to land on after login" }
|
|
611
|
+
},
|
|
612
|
+
"actions": [
|
|
613
|
+
{ "type": "goto", "value": "/login" },
|
|
614
|
+
{ "type": "type", "selector": "#email", "value": "{{email}}" },
|
|
615
|
+
{ "type": "type", "selector": "#password", "value": "{{password}}" },
|
|
616
|
+
{ "type": "click", "text": "Sign In" },
|
|
617
|
+
{ "type": "wait", "selector": "{{redirectTo}}" }
|
|
618
|
+
]
|
|
619
|
+
}
|
|
620
|
+
```
|
|
621
|
+
|
|
622
|
+
```json
|
|
623
|
+
// e2e/modules/auth-token.json — JWT injection (SPAs)
|
|
624
|
+
{
|
|
625
|
+
"$module": "auth-token",
|
|
626
|
+
"description": "Inject an auth token into browser storage",
|
|
627
|
+
"params": {
|
|
628
|
+
"token": { "required": true, "description": "JWT or session token" },
|
|
629
|
+
"storageKey": { "default": "accessToken", "description": "Storage key name" },
|
|
630
|
+
"storage": { "default": "local", "description": "local or session" },
|
|
631
|
+
"redirectTo": { "default": "/dashboard", "description": "Page to navigate to after injection" }
|
|
632
|
+
},
|
|
633
|
+
"actions": [
|
|
634
|
+
{ "type": "goto", "value": "/" },
|
|
635
|
+
{ "type": "set_storage", "value": "{{storageKey}}={{token}}", "selector": "{{#storage}}{{storage}}{{/storage}}" },
|
|
636
|
+
{ "type": "goto", "value": "{{redirectTo}}" }
|
|
637
|
+
]
|
|
638
|
+
}
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
Use in tests:
|
|
642
|
+
|
|
643
|
+
```json
|
|
644
|
+
// UI login
|
|
645
|
+
{ "$use": "login", "params": { "email": "admin@test.com", "password": "secret" } }
|
|
646
|
+
|
|
647
|
+
// Token injection
|
|
648
|
+
{ "$use": "auth-token", "params": { "token": "eyJhbG..." } }
|
|
649
|
+
|
|
650
|
+
// Token in sessionStorage, redirect to /settings
|
|
651
|
+
{ "$use": "auth-token", "params": { "token": "eyJhbG...", "storage": "session", "redirectTo": "/settings" } }
|
|
652
|
+
```
|
|
653
|
+
|
|
654
|
+
### Testing Different User Roles
|
|
655
|
+
|
|
656
|
+
Use separate tests (or the same module with different credentials) to test role-based access:
|
|
657
|
+
|
|
658
|
+
```json
|
|
659
|
+
[
|
|
660
|
+
{
|
|
661
|
+
"name": "admin-sees-settings",
|
|
662
|
+
"actions": [
|
|
663
|
+
{ "$use": "login", "params": { "email": "admin@test.com", "password": "admin-pass" } },
|
|
664
|
+
{ "type": "goto", "value": "/settings" },
|
|
665
|
+
{ "type": "assert_visible", "selector": ".admin-panel" }
|
|
666
|
+
]
|
|
667
|
+
},
|
|
668
|
+
{
|
|
669
|
+
"name": "viewer-cannot-access-settings",
|
|
670
|
+
"actions": [
|
|
671
|
+
{ "$use": "login", "params": { "email": "viewer@test.com", "password": "viewer-pass" } },
|
|
672
|
+
{ "type": "goto", "value": "/settings" },
|
|
673
|
+
{ "type": "assert_text", "text": "Access Denied" }
|
|
674
|
+
]
|
|
675
|
+
}
|
|
676
|
+
]
|
|
677
|
+
```
|
|
678
|
+
|
|
679
|
+
### Clearing Auth State
|
|
680
|
+
|
|
681
|
+
Each test runs in a **fresh browser context** (new connection to the Chrome pool), so cookies and storage are automatically clean. If you need to explicitly clear state mid-test:
|
|
682
|
+
|
|
683
|
+
```json
|
|
684
|
+
{ "type": "clear_cookies" }
|
|
685
|
+
```
|
|
686
|
+
|
|
687
|
+
This clears cookies, localStorage, and sessionStorage for the current origin.
|
|
688
|
+
|
|
689
|
+
### Quick Reference
|
|
690
|
+
|
|
691
|
+
| Auth type | Strategy | Key actions |
|
|
692
|
+
|-----------|----------|-------------|
|
|
693
|
+
| Username/password form | UI Login | `goto` + `type` + `click` in `beforeEach` |
|
|
694
|
+
| JWT in localStorage | Token Injection | `set_storage` in `beforeEach` |
|
|
695
|
+
| JWT in sessionStorage | Token Injection | `set_storage` with `selector: "session"` |
|
|
696
|
+
| Session cookies | Cookie | `evaluate` to set `document.cookie` |
|
|
697
|
+
| HttpOnly cookies | UI Login | Must go through login form |
|
|
698
|
+
| OAuth / SSO | Test bypass | App-specific test login endpoint |
|
|
699
|
+
| API auth headers | Header Override | `evaluate` to patch `fetch` |
|
|
700
|
+
| Config-level token | Config | `authToken` + `authStorageKey` in config |
|
|
701
|
+
|
|
702
|
+
---
|
|
703
|
+
|
|
248
704
|
## Reusable Modules
|
|
249
705
|
|
|
250
706
|
Extract common flows into parameterized modules:
|
|
251
707
|
|
|
252
708
|
```json
|
|
253
|
-
// e2e/modules/
|
|
709
|
+
// e2e/modules/login.json
|
|
254
710
|
{
|
|
255
|
-
"$module": "
|
|
256
|
-
"description": "
|
|
711
|
+
"$module": "login",
|
|
712
|
+
"description": "Log in via the UI login form",
|
|
257
713
|
"params": {
|
|
258
|
-
"
|
|
259
|
-
"
|
|
714
|
+
"email": { "required": true, "description": "User email" },
|
|
715
|
+
"password": { "required": true, "description": "User password" }
|
|
260
716
|
},
|
|
261
717
|
"actions": [
|
|
262
|
-
{ "type": "
|
|
263
|
-
{ "type": "
|
|
718
|
+
{ "type": "goto", "value": "/login" },
|
|
719
|
+
{ "type": "type", "selector": "#email", "value": "{{email}}" },
|
|
720
|
+
{ "type": "type", "selector": "#password", "value": "{{password}}" },
|
|
721
|
+
{ "type": "click", "text": "Sign In" },
|
|
722
|
+
{ "type": "wait", "value": "2000" }
|
|
264
723
|
]
|
|
265
724
|
}
|
|
266
725
|
```
|
|
@@ -271,7 +730,7 @@ Use in tests:
|
|
|
271
730
|
{
|
|
272
731
|
"name": "dashboard-loads",
|
|
273
732
|
"actions": [
|
|
274
|
-
{ "$use": "
|
|
733
|
+
{ "$use": "login", "params": { "email": "user@test.com", "password": "secret" } },
|
|
275
734
|
{ "type": "assert_text", "text": "Dashboard" }
|
|
276
735
|
]
|
|
277
736
|
}
|
|
@@ -455,7 +914,11 @@ The package ships as a **Claude Code plugin** — a single install that gives Cl
|
|
|
455
914
|
### Install as Plugin (recommended)
|
|
456
915
|
|
|
457
916
|
```bash
|
|
458
|
-
|
|
917
|
+
# 1. Add the marketplace (one-time)
|
|
918
|
+
claude plugin marketplace add fastslack/mtw-e2e-runner
|
|
919
|
+
|
|
920
|
+
# 2. Install the plugin
|
|
921
|
+
claude plugin install e2e-runner@matware
|
|
459
922
|
```
|
|
460
923
|
|
|
461
924
|
**What you get:**
|
|
@@ -465,7 +928,7 @@ claude plugin install npm:@matware/e2e-runner
|
|
|
465
928
|
| **13 MCP tools** | Run tests, create test files, capture screenshots, query network logs, manage dashboard, verify issues, query learnings |
|
|
466
929
|
| **Skill** | Teaches Claude the full e2e-runner workflow — how to combine tools, interpret results, debug failures, create tests |
|
|
467
930
|
| **3 Commands** | `/e2e-runner:run` — run & analyze tests<br>`/e2e-runner:create-test` — explore UI and create tests<br>`/e2e-runner:verify-issue <url>` — verify GitHub/GitLab bugs |
|
|
468
|
-
| **
|
|
931
|
+
| **3 Agents** | **test-analyzer** — diagnoses failures, analyzes flaky tests, drills into network errors<br>**test-creator** — explores UI, discovers selectors, designs and validates tests<br>**test-improver** — refactors verbose evaluate actions, extracts modules, adds waits/retries, eliminates hardcoded delays |
|
|
469
932
|
|
|
470
933
|
### Install MCP-only (alternative)
|
|
471
934
|
|
|
@@ -515,6 +978,69 @@ claude mcp add --transport stdio --scope user e2e-runner \
|
|
|
515
978
|
|
|
516
979
|
---
|
|
517
980
|
|
|
981
|
+
## OpenCode Integration
|
|
982
|
+
|
|
983
|
+
The package also supports [OpenCode](https://github.com/anomalyco/opencode) with native MCP server configuration, skills, and commands.
|
|
984
|
+
|
|
985
|
+
### Quick Setup
|
|
986
|
+
|
|
987
|
+
```bash
|
|
988
|
+
# 1. Install the package
|
|
989
|
+
npm install --save-dev @matware/e2e-runner
|
|
990
|
+
|
|
991
|
+
# 2. Copy OpenCode config to your project
|
|
992
|
+
cp node_modules/@matware/e2e-runner/opencode.json ./
|
|
993
|
+
|
|
994
|
+
# 3. Copy skills and commands (optional)
|
|
995
|
+
mkdir -p .opencode
|
|
996
|
+
cp -r node_modules/@matware/e2e-runner/.opencode/* .opencode/
|
|
997
|
+
|
|
998
|
+
# 4. Start the Chrome pool
|
|
999
|
+
npx e2e-runner pool start
|
|
1000
|
+
```
|
|
1001
|
+
|
|
1002
|
+
### What's Included
|
|
1003
|
+
|
|
1004
|
+
| Component | Description |
|
|
1005
|
+
|-----------|-------------|
|
|
1006
|
+
| **15 MCP tools** | Same tools as Claude Code — run tests, create files, screenshots, network logs, learnings, etc. |
|
|
1007
|
+
| **Skill** | `e2e-testing` — full workflow guidance with references |
|
|
1008
|
+
| **3 Commands** | `/run`, `/create-test`, `/verify-issue` |
|
|
1009
|
+
|
|
1010
|
+
### MCP Configuration
|
|
1011
|
+
|
|
1012
|
+
The `opencode.json` configures the MCP server as a local process:
|
|
1013
|
+
|
|
1014
|
+
```json
|
|
1015
|
+
{
|
|
1016
|
+
"mcp": {
|
|
1017
|
+
"e2e-runner": {
|
|
1018
|
+
"type": "local",
|
|
1019
|
+
"command": "node",
|
|
1020
|
+
"args": ["node_modules/@matware/e2e-runner/bin/mcp-server.js"],
|
|
1021
|
+
"cwd": "${workspaceFolder}"
|
|
1022
|
+
}
|
|
1023
|
+
}
|
|
1024
|
+
}
|
|
1025
|
+
```
|
|
1026
|
+
|
|
1027
|
+
For global installation, use the binary directly:
|
|
1028
|
+
|
|
1029
|
+
```json
|
|
1030
|
+
{
|
|
1031
|
+
"mcp": {
|
|
1032
|
+
"e2e-runner": {
|
|
1033
|
+
"type": "local",
|
|
1034
|
+
"command": "e2e-runner-mcp"
|
|
1035
|
+
}
|
|
1036
|
+
}
|
|
1037
|
+
}
|
|
1038
|
+
```
|
|
1039
|
+
|
|
1040
|
+
See [OPENCODE.md](OPENCODE.md) for full documentation on OpenCode integration.
|
|
1041
|
+
|
|
1042
|
+
---
|
|
1043
|
+
|
|
518
1044
|
## Network Error Handling
|
|
519
1045
|
|
|
520
1046
|
### Explicit Assertion
|