@matware/e2e-runner 1.3.1 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +4 -4
- package/.claude-plugin/plugin.json +2 -2
- package/LICENSE +1 -1
- package/README.md +491 -225
- package/agents/test-creator.md +4 -2
- package/agents/test-improver.md +7 -4
- package/bin/cli.js +93 -19
- package/package.json +4 -3
- package/skills/e2e-testing/SKILL.md +5 -3
- package/skills/e2e-testing/references/action-types.md +35 -18
- package/skills/e2e-testing/references/test-json-format.md +23 -0
- package/skills/e2e-testing/references/troubleshooting.md +2 -26
- package/src/actions.js +181 -15
- package/src/config.js +6 -0
- package/src/dashboard.js +185 -9
- package/src/db.js +26 -0
- package/src/mcp-tools.js +238 -69
- package/src/module-analysis.js +247 -0
- package/src/module-resolver.js +35 -2
- package/src/narrate.js +33 -1
- package/src/pool-manager.js +46 -1
- package/src/pool.js +177 -20
- package/src/runner.js +144 -19
- package/src/visual-diff.js +74 -4
- package/src/websocket.js +14 -3
- package/src/wizard.js +184 -0
- package/templates/build-dashboard.js +3 -0
- package/templates/dashboard/js/api.js +60 -3
- package/templates/dashboard/js/init.js +46 -0
- package/templates/dashboard/js/keyboard.js +8 -7
- package/templates/dashboard/js/quicksearch.js +277 -0
- package/templates/dashboard/js/state.js +61 -7
- package/templates/dashboard/js/toast.js +1 -1
- package/templates/dashboard/js/utils.js +23 -2
- package/templates/dashboard/js/view-live.js +235 -42
- package/templates/dashboard/js/view-runs.js +469 -42
- package/templates/dashboard/js/view-tests.js +157 -16
- package/templates/dashboard/js/view-tools.js +234 -0
- package/templates/dashboard/js/view-watch.js +2 -2
- package/templates/dashboard/js/websocket.js +33 -3
- package/templates/dashboard/styles/base.css +489 -53
- package/templates/dashboard/styles/components.css +736 -84
- package/templates/dashboard/styles/view-live.css +459 -78
- package/templates/dashboard/styles/view-runs.css +826 -177
- package/templates/dashboard/styles/view-tests.css +440 -77
- package/templates/dashboard/styles/view-tools.css +206 -0
- package/templates/dashboard/styles/view-watch.css +198 -41
- package/templates/dashboard/template.html +356 -58
- package/templates/dashboard.html +5354 -722
- package/templates/docker-compose-lightpanda.yml +7 -0
package/agents/test-creator.md
CHANGED
|
@@ -63,11 +63,12 @@ You are a specialist in creating robust E2E tests for web applications. You expl
|
|
|
63
63
|
|
|
64
64
|
### Form Interaction
|
|
65
65
|
- Standard input → `type` (clears first)
|
|
66
|
-
- React controlled input → `type_react`
|
|
67
|
-
- Dropdown select → `select` (native) or `
|
|
66
|
+
- React controlled input → `type_react` (optional `blur`, `waitAfter`)
|
|
67
|
+
- Dropdown select → `select` (native) or `select_combobox` (MUI Autocomplete/Select — opens, optional `filter`, picks `text` in one action)
|
|
68
68
|
- Checkbox/radio → `click`
|
|
69
69
|
- Clear field → `clear`
|
|
70
70
|
- Submit → `click` on submit button or `press` Enter
|
|
71
|
+
- Confirm in a modal → `click` with `text` + `scope: "dialog"` (add `last: true` if multiple matches)
|
|
71
72
|
|
|
72
73
|
### Storage
|
|
73
74
|
- Set localStorage key → `set_storage` with `value: "key=val"`
|
|
@@ -83,6 +84,7 @@ You are a specialist in creating robust E2E tests for web applications. You expl
|
|
|
83
84
|
### Waiting
|
|
84
85
|
- Element appears → `wait` with `selector`
|
|
85
86
|
- Text appears → `wait` with `text`
|
|
87
|
+
- Element/spinner/dialog disappears → `wait` with `gone` (e.g. `{ "type": "wait", "gone": ".MuiBackdrop-root" }`)
|
|
86
88
|
- Fixed delay (last resort) → `wait` with `value` (ms)
|
|
87
89
|
|
|
88
90
|
### Assertions
|
package/agents/test-improver.md
CHANGED
|
@@ -24,9 +24,9 @@ You are a specialist in refactoring and optimizing existing E2E tests without ch
|
|
|
24
24
|
|
|
25
25
|
- **Evaluate replacement**: Replace verbose `evaluate` actions with equivalent built-in actions (`type_react`, `click_option`, `assert_element_text`, etc.)
|
|
26
26
|
- **Duplication extraction**: Identify repeated action sequences across tests and extract them into reusable modules (`$use`)
|
|
27
|
-
- **Selector hardening**: Replace brittle selectors (nth-child, deep nesting, generated classes) with stable alternatives (`data-testid`, `id`, text-based)
|
|
27
|
+
- **Selector hardening**: Replace brittle selectors (nth-child, deep nesting, generated classes) with stable alternatives (`data-testid`, `id`, text-based). Applies to *interaction* selectors only — assertion selectors are treated as stable contracts (see Rules)
|
|
28
28
|
- **Flaky test stabilization**: Add `wait` actions, `retries`, and `serial: true` based on historical failure data from the learning system
|
|
29
|
-
- **Fixed delay elimination**: Replace hardcoded `wait` with ms values with
|
|
29
|
+
- **Fixed delay elimination**: Replace hardcoded `wait` with ms values with condition waits — `wait` on a `selector`/`text` to appear, or `wait` with `gone` to wait for a spinner/backdrop/dialog to disappear (e.g. `{ "type": "wait", "gone": ".MuiBackdrop-root" }`)
|
|
30
30
|
- **Visual verification**: Add `expect` fields to tests that lack visual verification
|
|
31
31
|
- **Serial marking**: Mark tests that share mutable state as `serial: true` to prevent race conditions
|
|
32
32
|
- **Hook extraction**: Move duplicated setup/teardown actions into `beforeEach`/`beforeAll` hooks
|
|
@@ -68,9 +68,11 @@ When you find an `evaluate` action, check if it matches one of these patterns
|
|
|
68
68
|
| `el.classList.contains(cls)` | `assert_class` with `selector` + `value` |
|
|
69
69
|
| `el.hasAttribute(attr)` or `el.getAttribute(attr)` | `assert_attribute` with `selector` + `value` |
|
|
70
70
|
| `document.querySelectorAll(sel).length` | `assert_count` with `selector` + `value` |
|
|
71
|
-
| Native value setter + `dispatchEvent(new Event('input'))` | `type_react` with `selector` + `value` |
|
|
72
|
-
| `querySelectorAll('[role="option"]')...click()` | `click_option` with `text` |
|
|
71
|
+
| Native value setter + `dispatchEvent(new Event('input'))` (single input) | `type_react` with `selector` + `value` (+ `blur:true` / `waitAfter` if it blurred/slept) |
|
|
72
|
+
| `querySelectorAll('[role="option"]')...click()` (no combobox open first) | `click_option` with `text` |
|
|
73
|
+
| Open combobox (focus/click input) + optional type filter + click matching option | `select_combobox` with `selector` + `text` (+ `filter`) |
|
|
73
74
|
| `MuiAutocomplete-root...input.focus()` | `focus_autocomplete` with `text` |
|
|
75
|
+
| Find a button by text inside `[role="dialog"]`/`.MuiDialog-root` and click (often the LAST one) | `click` with `text` + `scope: "dialog"` (+ `last: true`) |
|
|
74
76
|
| `querySelectorAll('button').filter(regex)...click()` | `click_regex` with `text` + optional `selector` + `value` |
|
|
75
77
|
| `querySelectorAll('[class*="Chip"]')...click()` | `click_chip` with `text` |
|
|
76
78
|
| `localStorage.setItem(key, val)` or `sessionStorage.setItem(...)` | `set_storage` with `value: "key=val"`, `selector: "session"` for session |
|
|
@@ -165,6 +167,7 @@ When extracting to a module, use `{{param}}` placeholders for values that vary b
|
|
|
165
167
|
4. **Preserve test ordering** — don't reorder tests within a suite. Numeric prefix ordering is intentional.
|
|
166
168
|
5. **Keep evaluates when no built-in exists** — if the evaluate does something that no built-in action covers (e.g., complex DOM manipulation, localStorage checks), leave it as-is.
|
|
167
169
|
6. **Prefer selector waits over fixed delays** — replace `{ "type": "wait", "value": "3000" }` with `{ "type": "wait", "selector": ".expected-element" }` when possible. Only keep fixed delays when there's genuinely no element to wait for.
|
|
170
|
+
7. **Assertion selectors are the contract** — interaction selectors may heal, but never retarget the `selector` of an `assert_*` action while hardening: that silently changes *what* the test verifies. If an assertion selector is genuinely broken, pin it to a stable `data-testid` and call it out in the summary instead of swapping it for whatever makes the test green.
|
|
168
171
|
|
|
169
172
|
## Output
|
|
170
173
|
|
package/bin/cli.js
CHANGED
|
@@ -21,7 +21,8 @@
|
|
|
21
21
|
* e2e-runner issue <url> --generate Generate test file via Claude API
|
|
22
22
|
* e2e-runner issue <url> --verify Generate + run + report bug status
|
|
23
23
|
* e2e-runner issue <url> --prompt Output the AI prompt (for piping)
|
|
24
|
-
* e2e-runner init
|
|
24
|
+
* e2e-runner init Interactive wizard to scaffold e2e/
|
|
25
|
+
* e2e-runner init --yes Scaffold with defaults (no prompts)
|
|
25
26
|
* e2e-runner --help Show help
|
|
26
27
|
* e2e-runner --version Show version
|
|
27
28
|
*/
|
|
@@ -34,6 +35,7 @@ import { loadConfig } from '../src/config.js';
|
|
|
34
35
|
import { startPool, stopPool, restartPool, connectToPool } from '../src/pool.js';
|
|
35
36
|
import { getPoolUrls, getAggregatedPoolStatus, waitForAnyPool, selectPool } from '../src/pool-manager.js';
|
|
36
37
|
import { runTestsParallel, loadTestFile, loadTestSuite, loadAllSuites, listSuites } from '../src/runner.js';
|
|
38
|
+
import { looksLikeBlankCapture } from '../src/actions.js';
|
|
37
39
|
import { generateReport, saveReport, printReport, persistRun, printInsights } from '../src/reporter.js';
|
|
38
40
|
import { startDashboard } from '../src/dashboard.js';
|
|
39
41
|
import { startWatch } from '../src/watch.js';
|
|
@@ -43,6 +45,7 @@ import { verifyIssue } from '../src/verify.js';
|
|
|
43
45
|
import { ensureProject, computeScreenshotHash, registerScreenshotHash } from '../src/db.js';
|
|
44
46
|
import { log, colors as C } from '../src/logger.js';
|
|
45
47
|
import { listModules } from '../src/module-resolver.js';
|
|
48
|
+
import { runInitWizard, renderConfig, getDefaultAnswers } from '../src/wizard.js';
|
|
46
49
|
import { getLearningsSummary, getFlakySummary, getSelectorStability, getPageHealth, getApiHealth, getErrorPatterns, getTestTrends } from '../src/learner-sqlite.js';
|
|
47
50
|
import { startNeo4j, stopNeo4j, getNeo4jStatus } from '../src/neo4j-pool.js';
|
|
48
51
|
import {
|
|
@@ -118,6 +121,8 @@ function parseCLIConfig() {
|
|
|
118
121
|
cliArgs.verificationStrictness = val;
|
|
119
122
|
}
|
|
120
123
|
}
|
|
124
|
+
if (getFlag('--driver')) cliArgs.cliDriverOverride = getFlag('--driver');
|
|
125
|
+
if (getFlag('--fallback-driver')) cliArgs.cliFallbackDriverOverride = getFlag('--fallback-driver');
|
|
121
126
|
return cliArgs;
|
|
122
127
|
}
|
|
123
128
|
|
|
@@ -148,6 +153,7 @@ ${C.bold}Usage:${C.reset}
|
|
|
148
153
|
e2e-runner capture <url> --selector <sel> Wait for selector before capture
|
|
149
154
|
e2e-runner capture <url> --delay <ms> Wait before capturing
|
|
150
155
|
e2e-runner capture <url> --filename <name> Custom filename
|
|
156
|
+
e2e-runner capture <url> --force Save even if the frame is blank
|
|
151
157
|
|
|
152
158
|
e2e-runner issue <url> Fetch issue and show details
|
|
153
159
|
e2e-runner issue <url> --generate Generate test file via Claude API
|
|
@@ -175,7 +181,10 @@ ${C.bold}Usage:${C.reset}
|
|
|
175
181
|
e2e-runner sync push Process sync queue (agent mode)
|
|
176
182
|
e2e-runner sync pull Pull runs from hub (agent mode)
|
|
177
183
|
|
|
178
|
-
e2e-runner init
|
|
184
|
+
e2e-runner init Interactive wizard to scaffold e2e/
|
|
185
|
+
e2e-runner init --yes Scaffold with defaults (CI / non-interactive)
|
|
186
|
+
Flags: --name, --base-url, --driver,
|
|
187
|
+
--pool-port, --concurrency, --no-sample
|
|
179
188
|
|
|
180
189
|
${C.bold}Options:${C.reset}
|
|
181
190
|
--base-url <url> App base URL (default: http://host.docker.internal:3000)
|
|
@@ -199,6 +208,9 @@ ${C.bold}Options:${C.reset}
|
|
|
199
208
|
--auth-login-endpoint <url> Auto-login: POST credentials to this URL to get auth token
|
|
200
209
|
--auth-token-path <path> Dot-path to token in auth response (default: token)
|
|
201
210
|
--verification-strictness <level> Visual verification: strict, moderate (default), lenient
|
|
211
|
+
--driver <name> Force pool driver for this run: browserless, cdp, lightpanda, obscura, steel
|
|
212
|
+
(overrides per-test "driver" field; useful for A/B benchmarks)
|
|
213
|
+
--fallback-driver <name> Explicit fallback if no pool with --driver is reachable (overrides per-test "fallbackDriver")
|
|
202
214
|
|
|
203
215
|
${C.bold}Watch Options:${C.reset}
|
|
204
216
|
--interval <time> Run interval: 15m, 1h, 30s (required for schedule mode)
|
|
@@ -220,6 +232,21 @@ async function cmdRun() {
|
|
|
220
232
|
const cliArgs = parseCLIConfig();
|
|
221
233
|
const config = await loadConfig(cliArgs);
|
|
222
234
|
config.triggeredBy = 'cli';
|
|
235
|
+
|
|
236
|
+
// Validate CLI driver overrides up-front (clearer error than waiting for first test)
|
|
237
|
+
if (config.cliDriverOverride || config.cliFallbackDriverOverride) {
|
|
238
|
+
const allowed = ['browserless', 'cdp', 'lightpanda', 'obscura', 'steel'];
|
|
239
|
+
for (const [flag, val] of [['--driver', config.cliDriverOverride], ['--fallback-driver', config.cliFallbackDriverOverride]]) {
|
|
240
|
+
if (val && !allowed.includes(val)) {
|
|
241
|
+
console.error(`${C.red}Invalid value for ${flag}: "${val}". Allowed: ${allowed.join(', ')}.${C.reset}`);
|
|
242
|
+
process.exit(1);
|
|
243
|
+
}
|
|
244
|
+
}
|
|
245
|
+
if (config.cliFallbackDriverOverride && !config.cliDriverOverride) {
|
|
246
|
+
console.error(`${C.red}--fallback-driver requires --driver.${C.reset}`);
|
|
247
|
+
process.exit(1);
|
|
248
|
+
}
|
|
249
|
+
}
|
|
223
250
|
let tests = [];
|
|
224
251
|
let hooks = {};
|
|
225
252
|
|
|
@@ -262,9 +289,32 @@ async function cmdRun() {
|
|
|
262
289
|
process.exit(1);
|
|
263
290
|
}
|
|
264
291
|
|
|
265
|
-
// Verify pool connectivity
|
|
292
|
+
// Verify pool connectivity — auto-start the Docker-managed pool if none is
|
|
293
|
+
// reachable, so first-time users don't need a separate `pool start` step.
|
|
266
294
|
log('🔌', `Checking Chrome Pool${poolUrls.length > 1 ? 's' : ''}...`);
|
|
267
|
-
const
|
|
295
|
+
const driverOpts = { poolDriver: config.poolDriver, maxSessions: config.maxSessions };
|
|
296
|
+
const _driver = config.poolDriver || 'auto';
|
|
297
|
+
const _dockerManaged = ['auto', 'browserless', 'lightpanda'].includes(_driver);
|
|
298
|
+
const _autoStart = config.autoStartPool !== false;
|
|
299
|
+
let pressure;
|
|
300
|
+
try {
|
|
301
|
+
pressure = await waitForAnyPool(poolUrls, 5000, driverOpts);
|
|
302
|
+
} catch {
|
|
303
|
+
if (_autoStart && _dockerManaged) {
|
|
304
|
+
log('🐳', `${C.dim}No pool detected — starting Chrome pool via Docker...${C.reset}`);
|
|
305
|
+
try {
|
|
306
|
+
startPool(config);
|
|
307
|
+
} catch (se) {
|
|
308
|
+
console.error(`${C.red}Could not auto-start the Chrome pool: ${se.message}${C.reset}`);
|
|
309
|
+
console.error(`${C.dim}Is Docker running? You can also start it manually: ${C.cyan}e2e-runner pool start${C.reset}`);
|
|
310
|
+
process.exit(1);
|
|
311
|
+
}
|
|
312
|
+
pressure = await waitForAnyPool(poolUrls, 45000, driverOpts);
|
|
313
|
+
} else {
|
|
314
|
+
console.error(`${C.red}No Chrome Pool available.${C.reset} Driver "${_driver}" is not Docker-managed — start your browser endpoint, then re-run.`);
|
|
315
|
+
process.exit(1);
|
|
316
|
+
}
|
|
317
|
+
}
|
|
268
318
|
log('✅', `Pool ready (${pressure.running}/${pressure.maxConcurrent} sessions, queued: ${pressure.queued})`);
|
|
269
319
|
|
|
270
320
|
// Wire up live progress to dashboard if running
|
|
@@ -387,10 +437,23 @@ async function cmdPool() {
|
|
|
387
437
|
}
|
|
388
438
|
}
|
|
389
439
|
|
|
390
|
-
function cmdInit() {
|
|
440
|
+
async function cmdInit() {
|
|
391
441
|
const cwd = process.cwd();
|
|
392
442
|
const templatesDir = path.join(__dirname, '..', 'templates');
|
|
393
443
|
|
|
444
|
+
const skipWizard = hasFlag('--yes') || hasFlag('-y') || hasFlag('--non-interactive');
|
|
445
|
+
const flagOverrides = {};
|
|
446
|
+
if (getFlag('--name') && typeof getFlag('--name') === 'string') flagOverrides.projectName = getFlag('--name');
|
|
447
|
+
if (getFlag('--base-url') && typeof getFlag('--base-url') === 'string') flagOverrides.baseUrl = getFlag('--base-url');
|
|
448
|
+
if (getFlag('--driver') && typeof getFlag('--driver') === 'string') flagOverrides.driver = getFlag('--driver');
|
|
449
|
+
if (getFlag('--pool-port') && typeof getFlag('--pool-port') === 'string') flagOverrides.poolPort = parseInt(getFlag('--pool-port'), 10);
|
|
450
|
+
if (getFlag('--concurrency') && typeof getFlag('--concurrency') === 'string') flagOverrides.concurrency = parseInt(getFlag('--concurrency'), 10);
|
|
451
|
+
if (hasFlag('--no-sample')) flagOverrides.includeSampleTest = false;
|
|
452
|
+
|
|
453
|
+
const answers = skipWizard
|
|
454
|
+
? { ...getDefaultAnswers(cwd), ...flagOverrides }
|
|
455
|
+
: await runInitWizard(cwd, flagOverrides);
|
|
456
|
+
|
|
394
457
|
// Create directory structure
|
|
395
458
|
const dirs = [
|
|
396
459
|
path.join(cwd, 'e2e', 'tests'),
|
|
@@ -405,22 +468,24 @@ function cmdInit() {
|
|
|
405
468
|
}
|
|
406
469
|
}
|
|
407
470
|
|
|
408
|
-
//
|
|
471
|
+
// Write generated config
|
|
409
472
|
const configDest = path.join(cwd, 'e2e.config.js');
|
|
410
473
|
if (!fs.existsSync(configDest)) {
|
|
411
|
-
fs.
|
|
474
|
+
fs.writeFileSync(configDest, renderConfig(answers));
|
|
412
475
|
log('📄', 'Created e2e.config.js');
|
|
413
476
|
} else {
|
|
414
477
|
log('⏭️', 'e2e.config.js already exists, skipping');
|
|
415
478
|
}
|
|
416
479
|
|
|
417
480
|
// Copy sample test
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
fs.
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
481
|
+
if (answers.includeSampleTest) {
|
|
482
|
+
const testDest = path.join(cwd, 'e2e', 'tests', 'sample.json');
|
|
483
|
+
if (!fs.existsSync(testDest)) {
|
|
484
|
+
fs.copyFileSync(path.join(templatesDir, 'sample-test.json'), testDest);
|
|
485
|
+
log('📄', 'Created e2e/tests/sample.json');
|
|
486
|
+
} else {
|
|
487
|
+
log('⏭️', 'e2e/tests/sample.json already exists, skipping');
|
|
488
|
+
}
|
|
424
489
|
}
|
|
425
490
|
|
|
426
491
|
// Create .gitkeep
|
|
@@ -455,9 +520,9 @@ ${C.bold}${C.green}E2E structure created!${C.reset}
|
|
|
455
520
|
|
|
456
521
|
${C.bold}Next steps:${C.reset}
|
|
457
522
|
1. Edit ${C.cyan}e2e.config.js${C.reset} with your app URL
|
|
458
|
-
2.
|
|
459
|
-
|
|
460
|
-
|
|
523
|
+
2. Run your tests: ${C.cyan}e2e-runner run --all${C.reset} ${C.dim}(starts Chrome automatically)${C.reset}
|
|
524
|
+
|
|
525
|
+
${C.dim}That's it — the runner spins up the Chrome pool for you on first run.${C.reset}
|
|
461
526
|
`);
|
|
462
527
|
}
|
|
463
528
|
|
|
@@ -483,7 +548,7 @@ async function cmdDashboard() {
|
|
|
483
548
|
async function cmdCapture() {
|
|
484
549
|
const url = args[1];
|
|
485
550
|
if (!url || url.startsWith('--')) {
|
|
486
|
-
console.error(`${C.red}Usage: e2e-runner capture <url> [--filename <name>] [--full-page] [--selector <sel>] [--delay <ms>]${C.reset}`);
|
|
551
|
+
console.error(`${C.red}Usage: e2e-runner capture <url> [--filename <name>] [--full-page] [--selector <sel>] [--delay <ms>] [--force]${C.reset}`);
|
|
487
552
|
process.exit(1);
|
|
488
553
|
}
|
|
489
554
|
|
|
@@ -529,7 +594,16 @@ async function cmdCapture() {
|
|
|
529
594
|
|
|
530
595
|
const screenshotPath = path.join(config.screenshotsDir, filename);
|
|
531
596
|
const fullPage = hasFlag('--full-page');
|
|
532
|
-
await page.screenshot({
|
|
597
|
+
const captureBuf = await page.screenshot({ fullPage });
|
|
598
|
+
|
|
599
|
+
// Blank frame (uniform color — page never rendered): skip the save
|
|
600
|
+
// unless the user explicitly forces it.
|
|
601
|
+
if (!hasFlag('--force') && looksLikeBlankCapture(captureBuf, 'png')) {
|
|
602
|
+
log('⚠️', `${C.yellow}Capture skipped:${C.reset} page rendered a blank (uniform-color) frame — nothing saved. Use ${C.dim}--force${C.reset} to save anyway.`);
|
|
603
|
+
console.log('');
|
|
604
|
+
return;
|
|
605
|
+
}
|
|
606
|
+
fs.writeFileSync(screenshotPath, captureBuf);
|
|
533
607
|
|
|
534
608
|
// Register hash in SQLite
|
|
535
609
|
const cwd = process.cwd();
|
|
@@ -1138,7 +1212,7 @@ async function main() {
|
|
|
1138
1212
|
break;
|
|
1139
1213
|
|
|
1140
1214
|
case 'init':
|
|
1141
|
-
cmdInit();
|
|
1215
|
+
await cmdInit();
|
|
1142
1216
|
break;
|
|
1143
1217
|
|
|
1144
1218
|
default:
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@matware/e2e-runner",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.5.1",
|
|
4
4
|
"mcpName": "io.github.fastslack/e2e-runner",
|
|
5
5
|
"description": "E2E test runner using Chrome Pool (browserless/chrome) with parallel execution",
|
|
6
6
|
"type": "module",
|
|
@@ -39,7 +39,7 @@
|
|
|
39
39
|
"github-issues",
|
|
40
40
|
"ai-testing"
|
|
41
41
|
],
|
|
42
|
-
"author": "Matware",
|
|
42
|
+
"author": "Matias Aguirre (Matware)",
|
|
43
43
|
"license": "Apache-2.0",
|
|
44
44
|
"repository": {
|
|
45
45
|
"type": "git",
|
|
@@ -52,7 +52,8 @@
|
|
|
52
52
|
"puppeteer-core": "^24.0.0"
|
|
53
53
|
},
|
|
54
54
|
"scripts": {
|
|
55
|
-
"build:dashboard": "node templates/build-dashboard.js"
|
|
55
|
+
"build:dashboard": "node templates/build-dashboard.js",
|
|
56
|
+
"prepublishOnly": "node templates/build-dashboard.js"
|
|
56
57
|
},
|
|
57
58
|
"engines": {
|
|
58
59
|
"node": ">=20.0.0"
|
|
@@ -9,7 +9,7 @@ description: Create, run, and debug JSON-driven E2E browser tests with Chrome po
|
|
|
9
9
|
|
|
10
10
|
`@matware/e2e-runner` is a JSON-driven E2E test runner. Tests are defined as JSON files with sequential browser actions — no JavaScript test code. Tests run in parallel against a Chrome pool (browserless/chrome via Docker) using Puppeteer.
|
|
11
11
|
|
|
12
|
-
**Key capabilities:**
|
|
12
|
+
**Key capabilities:** 17 MCP tools for running tests, creating test files, capturing screenshots, analyzing network traffic, verifying GitHub/GitLab issues, and querying a learning system for stability insights.
|
|
13
13
|
|
|
14
14
|
## Prerequisites
|
|
15
15
|
|
|
@@ -72,8 +72,9 @@ Use `e2e_create_test` to write test files. Use `e2e_create_module` for reusable
|
|
|
72
72
|
### Key Action Patterns
|
|
73
73
|
|
|
74
74
|
- **Navigation**: `goto` (full page load), `navigate` (SPA-friendly, non-blocking)
|
|
75
|
-
- **Interaction**: `click` (selector or text), `type`/`fill`, `select`, `press`, `hover`, `scroll`
|
|
76
|
-
- **React/MUI**: `type_react` (controlled inputs), `click_option`, `focus_autocomplete`, `click_chip`, `click_regex`
|
|
75
|
+
- **Interaction**: `click` (selector or text; text mode also takes `scope:"dialog"`, `visible:true`, `last:true`), `type`/`fill`, `select`, `press`, `hover`, `scroll`
|
|
76
|
+
- **React/MUI**: `type_react` (controlled inputs; optional `blur`, `waitAfter`), `click_option`, `select_combobox` (open+filter+pick MUI Autocomplete/Select in one action), `focus_autocomplete`, `click_chip`, `click_regex`
|
|
77
|
+
- **Waiting**: prefer conditions over sleeps — `wait` takes `selector`/`text` (appear), `gone` (disappear, e.g. spinner/closing dialog), or `value` (fixed ms, last resort); `wait_network_idle`
|
|
77
78
|
- **Assertions**: `assert_text` (page-wide), `assert_element_text` (scoped), `assert_url`, `assert_visible`, `assert_not_visible`, `assert_count`, `assert_attribute`, `assert_class`, `assert_input_value`, `assert_matches`
|
|
78
79
|
- **Extraction**: `get_text` (non-assertion, returns element text), `screenshot`
|
|
79
80
|
- **Advanced**: `evaluate` (run JS in browser), `assert_no_network_errors`, `clear_cookies`
|
|
@@ -158,6 +159,7 @@ Start/stop the web dashboard with `e2e_dashboard_start` / `e2e_dashboard_stop` f
|
|
|
158
159
|
4. **`evaluate` is strict** — Returns starting with `FAIL:`/`ERROR:` or returning `false` will fail the test. Prefer granular assertion actions over `evaluate` with inline JS.
|
|
159
160
|
5. **Serial tests** — Mark tests with `"serial": true` if they share mutable state. They run after all parallel tests.
|
|
160
161
|
6. **Action retries** — Use `"retries": N` on individual actions for flaky selectors, or globally via config.
|
|
162
|
+
7. **Assertion selectors are the contract** — when fixing flaky tests, heal *interaction* selectors freely, but never retarget an `assert_*` selector to make a test pass: pin assertion selectors to stable `data-testid`s.
|
|
161
163
|
|
|
162
164
|
## References
|
|
163
165
|
|
|
@@ -13,7 +13,7 @@ Complete catalog of all action types supported by @matware/e2e-runner.
|
|
|
13
13
|
|
|
14
14
|
| Action | Fields | Description |
|
|
15
15
|
|--------|--------|-------------|
|
|
16
|
-
| `click` | `selector` OR `text` | Click by CSS selector or by visible text content. Text search covers: `button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"], [role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6, dd, dt`. |
|
|
16
|
+
| `click` | `selector` OR `text` | Click by CSS selector or by visible text content. Text search covers: `button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"], [role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6, dd, dt`. Optional text-mode refinements: `scope: "dialog"` (only match inside an open `[role="dialog"]`/`.MuiDialog-root`), `visible: true` (skip hidden/zero-size matches — implied by `scope:dialog`), `last: true` (click the LAST match instead of the first). Prefer these over hand-rolled `evaluate` button-by-text scans. |
|
|
17
17
|
| `type` / `fill` | `selector`, `value` | Triple-clicks to select all, then Backspace to clear, then types with 20ms delay per character. |
|
|
18
18
|
| `select` | `selector`, `value` | Select an `<option>` value in a `<select>` element. |
|
|
19
19
|
| `clear` | `selector` | Triple-click + Backspace to clear an input field. |
|
|
@@ -25,9 +25,10 @@ Complete catalog of all action types supported by @matware/e2e-runner.
|
|
|
25
25
|
|
|
26
26
|
| Action | Fields | Description |
|
|
27
27
|
|--------|--------|-------------|
|
|
28
|
-
| `type_react` | `selector`, `value` | Types into React controlled inputs using native value setter.
|
|
28
|
+
| `type_react` | `selector`, `value`, `blur` (optional), `waitAfter` (optional ms) | Types into React controlled inputs using native value setter. Focuses, then dispatches `input` + `change` events so React state updates. Supports `<input>` and `<textarea>`. `blur: true` commits on blur (for fields that validate on blur); `waitAfter: "<ms>"` waits after (e.g. for debounced autocomplete). Prefer over inline `setNativeValue` evaluates. |
|
|
29
29
|
| `click_regex` | `text` (regex), `selector` (optional), `value` (`"last"` optional) | Click element whose textContent matches regex (case-insensitive). Default: first match. `value: "last"` for last match. `selector` scopes the search. |
|
|
30
30
|
| `click_option` | `text` | Click a `[role="option"]` element by text — for autocomplete/select dropdowns. Waits for option to appear. |
|
|
31
|
+
| `select_combobox` | `selector` (optional, default `input[role='combobox']`), `text` (option to pick), `filter` (optional typed text), `openWait`/`filterWait`/`waitAfter` (optional ms) | Open a MUI Autocomplete/Select, optionally type `filter` to narrow, then click the option matching `text` (case-insensitive substring). Falls back across `[role="option"]`, `.MuiAutocomplete-option`, `li.MuiMenuItem-root`. Replaces the verbose open-input + setNativeValue + scan-options `evaluate` pattern. |
|
|
31
32
|
| `focus_autocomplete` | `text` (label text) | Focus an autocomplete input by label. Supports MUI `.MuiAutocomplete-root` and `[role="combobox"]`. |
|
|
32
33
|
| `click_chip` | `text` | Click a chip/tag element by text. Searches `[class*="Chip"]`, `[class*="chip"]`, `[data-chip]`. |
|
|
33
34
|
|
|
@@ -46,11 +47,25 @@ Complete catalog of all action types supported by @matware/e2e-runner.
|
|
|
46
47
|
| `click_menu_item` | `text` (menu item text), `selector` (scope, optional) | Click a menu item by text. Searches `[role="menuitem"]`, `[role="menuitemradio"]`, `[role="menuitemcheckbox"]`, `.dropdown-item`, `.menu-item`, `[class*="MenuItem"]`, `[role="menu"] > li`. Waits for element to appear. |
|
|
47
48
|
| `click_in_context` | `text` (container text), `selector` (child to click) | Find the smallest container whose text includes `text`, then click the `selector` child within it. Containers: `section`, `article`, `[class*="card"]`, `li`, `tr`, `div[class]`, etc. Both fields required. |
|
|
48
49
|
|
|
50
|
+
## Multi-Tab
|
|
51
|
+
|
|
52
|
+
All subsequent actions run in the active tab. The runner manages a tab registry keyed by label.
|
|
53
|
+
|
|
54
|
+
| Action | Fields | Description |
|
|
55
|
+
|--------|--------|-------------|
|
|
56
|
+
| `open_tab` | `value` (URL), `text` (label, optional) | Open a new tab and navigate (relative to `baseUrl` or absolute). Label defaults to `tab-<n>`. The new tab becomes active. |
|
|
57
|
+
| `switch_tab` | `value` | Switch active tab by label (exact), numeric index, or title/URL match (regex or substring). `"default"` returns to the original tab. |
|
|
58
|
+
| `wait_for_tab` | `text` (label, optional), `timeout` | Wait for a tab/popup opened by the app (`window.open`, `target="_blank"`) and make it active. Use right after the action that triggers the popup. |
|
|
59
|
+
| `assert_tab_count` | `value` | Assert number of open tabs: exact (`"2"`) or operators (`">=2"`). |
|
|
60
|
+
| `close_tab` | `value` (label, optional) | Close the current (or named) tab and switch back to the last remaining one. Cannot close `default` while other tabs are open. |
|
|
61
|
+
|
|
49
62
|
## Assertions
|
|
50
63
|
|
|
51
64
|
| Action | Fields | Description |
|
|
52
65
|
|--------|--------|-------------|
|
|
53
66
|
| `assert_text` | `text` | Check entire page body contains text (substring match). |
|
|
67
|
+
| `assert_no_text` | `text` | Check text does NOT appear anywhere in the page body. Opposite of `assert_text`. |
|
|
68
|
+
| `assert_text_in` | `selector`, `text`, `value` (`"exact"` optional) | Check text inside a scoped container. Joins `textContent` from all matching elements. Default: case-insensitive regex; with `value: "exact"`: case-sensitive substring. |
|
|
54
69
|
| `assert_element_text` | `selector`, `text`, `value` (`"exact"` optional) | Check specific element's `textContent`. Default: substring match. With `value: "exact"`: strict `trim() ===` comparison. |
|
|
55
70
|
| `assert_url` | `value` | Check current URL. Path-only (`/dashboard`) compares pathname. Full URL does substring match. |
|
|
56
71
|
| `assert_visible` | `selector` | Element exists and is visible (`display`, `visibility`, `opacity` checks). |
|
|
@@ -61,11 +76,14 @@ Complete catalog of all action types supported by @matware/e2e-runner.
|
|
|
61
76
|
| `assert_input_value` | `selector`, `value` | Checks `element.value.includes(value)` on input/select/textarea. |
|
|
62
77
|
| `assert_matches` | `selector`, `value` (regex) | Tests element's `textContent` against `new RegExp(value)`. |
|
|
63
78
|
| `assert_no_network_errors` | — | Checks accumulated `requestfailed` events during the test. Fails with error details if any exist. |
|
|
79
|
+
| `assert_visual` | `value` (golden image filename), `selector` (optional element scope), `text` (max diff fraction, e.g. `"0.02"`), plus `fullPage`, `maskRegions`, `threshold` | Visual regression against a golden reference image (`goldenDir`, default `{screenshotsDir}/golden`). First run saves the golden and passes; later runs fail if more pixels differ than the max diff (default 2%) and write a diff image. `maskRegions: [{x,y,width,height}]` ignores dynamic areas (timestamps, avatars). |
|
|
64
80
|
|
|
65
81
|
### Assertion Disambiguation
|
|
66
82
|
|
|
67
83
|
- **`assert_text`** → searches the **entire page body** (substring)
|
|
84
|
+
- **`assert_no_text`** → asserts text is **absent** from the page body (do NOT use `assert_not_visible` with `text` — it requires `selector`)
|
|
68
85
|
- **`assert_element_text`** → checks a **specific element** (substring, or exact with `value: "exact"`)
|
|
86
|
+
- **`assert_text_in`** → checks text inside a **scoped container** (regex by default)
|
|
69
87
|
- **`assert_matches`** → checks a specific element against a **regex** pattern
|
|
70
88
|
- **`assert_input_value`** → reads the `.value` property (for form fields)
|
|
71
89
|
|
|
@@ -75,7 +93,7 @@ Complete catalog of all action types supported by @matware/e2e-runner.
|
|
|
75
93
|
|--------|--------|-------------|
|
|
76
94
|
| `get_text` | `selector` | Returns `{ value: textContent.trim() }`. Non-assertion — never fails. |
|
|
77
95
|
| `screenshot` | `value` (filename, optional) | Captures screenshot. Filename gets timestamp suffix for uniqueness. |
|
|
78
|
-
| `wait` | `selector` OR `text` OR `value` (ms) |
|
|
96
|
+
| `wait` | `selector` OR `text` OR `gone` OR `value` (ms) | Prefer **conditions over fixed sleeps**: `{ selector }` waits for it to appear, `{ text }` waits for text to appear, **`{ gone: "<css>" }`** waits until a selector disappears/hides (spinner, closing dialog), `{ gone: true, selector|text }` is the explicit form, `{ value: "<ms>" }` is a fixed delay (last resort). Replacing `wait` sleeps with `gone`/`selector` makes suites faster and less flaky. |
|
|
79
97
|
| `wait_network_idle` | `value` (idle ms, default 500), `timeout` (max wait ms, default 30000) | Waits for all network requests to complete. Uses Puppeteer's `page.waitForNetworkIdle()`. Useful after SPA page transitions or data loading. |
|
|
80
98
|
| `evaluate` | `value` (JS code) | Run JavaScript in browser context. See **Strict Evaluate** below. |
|
|
81
99
|
| `clear_cookies` | `value` (origin, optional) | Clears cookies, localStorage, sessionStorage for origin. |
|
|
@@ -107,31 +125,30 @@ Delay between retries: `actionRetryDelay` config (default 500ms).
|
|
|
107
125
|
### React input + autocomplete flow
|
|
108
126
|
```json
|
|
109
127
|
{ "type": "focus_autocomplete", "text": "Category" },
|
|
110
|
-
{ "type": "type_react", "selector": "#category-input", "value": "Electr" },
|
|
128
|
+
{ "type": "type_react", "selector": "#category-input", "value": "Electr", "waitAfter": "400" },
|
|
111
129
|
{ "type": "click_option", "text": "Electronics" }
|
|
112
130
|
```
|
|
113
131
|
|
|
114
|
-
###
|
|
132
|
+
### MUI combobox in one action (open + filter + pick)
|
|
115
133
|
```json
|
|
116
|
-
{ "type": "
|
|
134
|
+
{ "type": "select_combobox", "selector": "[data-cy='specialty'] input", "filter": "cardio", "text": "Cardiología" }
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
### Condition waits instead of fixed sleeps (faster, less flaky)
|
|
138
|
+
```json
|
|
139
|
+
{ "type": "click", "text": "Guardar" },
|
|
140
|
+
{ "type": "wait", "gone": ".MuiBackdrop-root" },
|
|
141
|
+
{ "type": "wait", "selector": "[data-testid='saved-banner']" }
|
|
117
142
|
```
|
|
118
143
|
|
|
119
|
-
###
|
|
144
|
+
### Click a button inside an open dialog (no evaluate needed)
|
|
120
145
|
```json
|
|
121
|
-
{ "type": "
|
|
122
|
-
{ "type": "assert_attribute", "selector": "button.submit", "value": "disabled" },
|
|
123
|
-
{ "type": "assert_class", "selector": ".nav-item:first-child", "value": "active" },
|
|
124
|
-
{ "type": "assert_input_value", "selector": "#email", "value": "user@example.com" },
|
|
125
|
-
{ "type": "assert_matches", "selector": ".phone", "value": "\\d{3}-\\d{3}-\\d{4}" },
|
|
126
|
-
{ "type": "assert_count", "selector": ".table-row", "value": ">3" }
|
|
146
|
+
{ "type": "click", "text": "Iniciar encuentro", "scope": "dialog", "last": true }
|
|
127
147
|
```
|
|
128
148
|
|
|
129
|
-
###
|
|
149
|
+
### Regex click (last match)
|
|
130
150
|
```json
|
|
131
|
-
{ "type": "
|
|
132
|
-
{ "type": "assert_storage", "value": "authToken" },
|
|
133
|
-
{ "type": "set_storage", "value": "theme=dark", "selector": "session" },
|
|
134
|
-
{ "type": "assert_storage", "value": "theme=dark", "selector": "session" }
|
|
151
|
+
{ "type": "click_regex", "text": "add to cart", "selector": "button", "value": "last" }
|
|
135
152
|
```
|
|
136
153
|
|
|
137
154
|
### Icon, menu, and contextual clicks
|
|
@@ -113,6 +113,29 @@ Module definition (in `e2e/modules/auth-login.json`):
|
|
|
113
113
|
}
|
|
114
114
|
```
|
|
115
115
|
|
|
116
|
+
### Composing modules (nested `$use` + parameter forwarding)
|
|
117
|
+
|
|
118
|
+
A module can `$use` other modules, and **forward its own params/defaults** into the
|
|
119
|
+
nested call's `params` block. Placeholders in a nested `params` value are resolved
|
|
120
|
+
against the outer module's scope before the inner module runs:
|
|
121
|
+
|
|
122
|
+
```json
|
|
123
|
+
{
|
|
124
|
+
"$module": "login-and-open",
|
|
125
|
+
"params": {
|
|
126
|
+
"patientId": { "required": true },
|
|
127
|
+
"email": { "required": false, "default": "admin@test.com" }
|
|
128
|
+
},
|
|
129
|
+
"actions": [
|
|
130
|
+
{ "$use": "auth-login", "params": { "email": "{{email}}", "password": "secret" } },
|
|
131
|
+
{ "$use": "open-patient", "params": { "id": "{{patientId}}" } }
|
|
132
|
+
]
|
|
133
|
+
}
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Cycles are detected and rejected. Action types are validated **after** all `$use`
|
|
137
|
+
references are expanded.
|
|
138
|
+
|
|
116
139
|
## Suite Naming & Ordering
|
|
117
140
|
|
|
118
141
|
Files can have numeric prefixes for execution order:
|
|
@@ -193,32 +193,8 @@ The `KNOWN_ACTION_TYPES` Set in `src/actions.js` is the single source of truth.
|
|
|
193
193
|
|
|
194
194
|
## Screenshot Hashes
|
|
195
195
|
|
|
196
|
-
Every screenshot
|
|
197
|
-
|
|
198
|
-
**Flow**: screenshot saved on disk → `saveRun()` registers hash in SQLite `screenshot_hashes` table → dashboard shows `[ss:XXXXXXXX]` badge (click to copy) → user pastes hash in Claude Code → `e2e_screenshot` MCP tool looks up hash, reads file, returns the image.
|
|
199
|
-
|
|
200
|
-
- Hashes are registered inside the `saveRun()` transaction (covers action, error, verification, and baseline screenshots)
|
|
201
|
-
- The `ss:` prefix is optional when calling `e2e_screenshot` — stripped during lookup
|
|
202
|
-
- Dashboard computes hashes client-side (Web Crypto) for the Live view (before `persistRun()` writes to DB)
|
|
203
|
-
- Run detail API (`/api/db/runs/:id`) includes `screenshotHashes` map per test result
|
|
204
|
-
- Dashboard endpoint `/api/screenshot-hash/:hash` serves the image by hash
|
|
205
|
-
- Dashboard Screenshots view has a **search bar** — type a hash to find and display the screenshot
|
|
196
|
+
Every screenshot gets a short hash like `ss:a3f2b1c9` (deterministic, from its file path). The dashboard shows it as a click-to-copy badge; paste it (the `ss:` prefix is optional) and `e2e_screenshot` returns the image. The dashboard Screenshots view also has a hash search bar.
|
|
206
197
|
|
|
207
198
|
## Web Dashboard
|
|
208
199
|
|
|
209
|
-
|
|
210
|
-
**`templates/dashboard.html`** — SPA, dark theme, vanilla JS, safe DOM (textContent + createEl helper).
|
|
211
|
-
|
|
212
|
-
**Features:**
|
|
213
|
-
- Live test execution with WebSocket updates
|
|
214
|
-
- Run history with inline detail expansion
|
|
215
|
-
- Screenshots gallery with hash badges and hash search
|
|
216
|
-
- Network request logs with clickable expandable rows (full request/response detail)
|
|
217
|
-
- Pool status monitoring
|
|
218
|
-
- Multi-project support via project selector
|
|
219
|
-
- Variables tab with masked values, inline edit, add, and delete
|
|
220
|
-
|
|
221
|
-
**CLI:** `e2e-runner dashboard [--port 8484]`
|
|
222
|
-
**MCP tools:** `e2e_dashboard_start`, `e2e_dashboard_stop`
|
|
223
|
-
|
|
224
|
-
Config defaults: `dashboardPort: 8484`, `maxHistoryRuns: 100`
|
|
200
|
+
Start with `e2e_dashboard_start` (or `e2e-runner dashboard [--port 8484]`), stop with `e2e_dashboard_stop`. Provides live execution (WebSocket), run history, screenshot gallery + hash search, expandable network logs, pool status, multi-project selector, and a variables tab.
|