@matware/e2e-runner 1.1.1 → 1.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/.claude-plugin/plugin.json +9 -0
  2. package/.mcp.json +9 -0
  3. package/README.md +475 -307
  4. package/agents/test-analyzer.md +81 -0
  5. package/agents/test-creator.md +102 -0
  6. package/agents/test-improver.md +140 -0
  7. package/bin/cli.js +194 -6
  8. package/commands/create-test.md +50 -0
  9. package/commands/run.md +49 -0
  10. package/commands/verify-issue.md +63 -0
  11. package/package.json +10 -2
  12. package/skills/e2e-testing/SKILL.md +166 -0
  13. package/skills/e2e-testing/references/action-types.md +100 -0
  14. package/skills/e2e-testing/references/test-json-format.md +159 -0
  15. package/skills/e2e-testing/references/troubleshooting.md +182 -0
  16. package/src/actions.js +273 -18
  17. package/src/ai-generate.js +87 -7
  18. package/src/config.js +28 -0
  19. package/src/dashboard.js +156 -6
  20. package/src/db.js +207 -13
  21. package/src/index.js +9 -3
  22. package/src/learner-markdown.js +177 -0
  23. package/src/learner-neo4j.js +255 -0
  24. package/src/learner-sqlite.js +354 -0
  25. package/src/learner.js +413 -0
  26. package/src/mcp-tools.js +448 -18
  27. package/src/module-resolver.js +273 -0
  28. package/src/narrate.js +225 -0
  29. package/src/neo4j-pool.js +124 -0
  30. package/src/reporter.js +35 -2
  31. package/src/runner.js +120 -46
  32. package/src/verify.js +5 -3
  33. package/templates/build-dashboard.js +28 -0
  34. package/templates/dashboard/app.js +1152 -0
  35. package/templates/dashboard/styles.css +413 -0
  36. package/templates/dashboard/template.html +201 -0
  37. package/templates/dashboard.html +964 -378
  38. package/templates/docker-compose-neo4j.yml +19 -0
  39. package/templates/e2e.config.js +3 -0
@@ -0,0 +1,50 @@
1
+ ---
2
+ description: Create a new E2E test by exploring the UI and designing test actions
3
+ user_invocable: true
4
+ allowed_tools:
5
+ - mcp__e2e-runner__e2e_pool_status
6
+ - mcp__e2e-runner__e2e_capture
7
+ - mcp__e2e-runner__e2e_list
8
+ - mcp__e2e-runner__e2e_create_test
9
+ - mcp__e2e-runner__e2e_create_module
10
+ - mcp__e2e-runner__e2e_run
11
+ - mcp__e2e-runner__e2e_screenshot
12
+ - Read
13
+ - Grep
14
+ - Glob
15
+ ---
16
+
17
+ # Create E2E Test
18
+
19
+ Help the user create a new E2E test file by exploring the application and designing appropriate test actions.
20
+
21
+ ## Workflow
22
+
23
+ 1. **Understand the goal** — Ask the user what they want to test if not already specified. Identify the page(s), user flow, and expected outcomes.
24
+
25
+ 2. **Check pool** — Call `e2e_pool_status` to ensure the Chrome pool is available.
26
+
27
+ 3. **Explore the UI** — Use `e2e_capture` to screenshot the target page(s). This helps understand the current state of the UI, available elements, and layout.
28
+
29
+ 4. **Check existing tests** — Call `e2e_list` to see what test suites already exist. Read relevant existing test files with `Read` to follow conventions and avoid duplication.
30
+
31
+ 5. **Explore source code** (optional) — If needed, use `Grep` and `Read` to find selectors, form field IDs, API endpoints, or component structure in the application source code.
32
+
33
+ 6. **Design the test** — Based on UI exploration and source code analysis, design the test actions:
34
+ - Use the most specific selectors available (data-testid > id > class > text)
35
+ - Prefer granular assertion actions over `evaluate`
36
+ - Use framework-aware actions for React/MUI (`type_react`, `click_option`, `focus_autocomplete`)
37
+ - Add `wait` actions before assertions on dynamic content
38
+ - Add `assert_no_network_errors` after critical page loads
39
+ - Consider adding an `expect` field for visual verification
40
+
41
+ 7. **Create the test** — Call `e2e_create_test` with the designed test structure. Consider creating reusable modules with `e2e_create_module` for repeated sequences (auth, navigation).
42
+
43
+ 8. **Validate** — Run the newly created test with `e2e_run` using the `suite` parameter. Analyze results and iterate if needed.
44
+
45
+ ## Arguments
46
+
47
+ The user may provide:
48
+ - A test name: `/e2e-runner:create-test login-flow`
49
+ - A description of what to test: `/e2e-runner:create-test test the checkout process`
50
+ - A URL to start from: `/e2e-runner:create-test http://localhost:3000/checkout`
@@ -0,0 +1,49 @@
1
+ ---
2
+ description: Run E2E tests and analyze results with screenshots and network drill-down
3
+ user_invocable: true
4
+ allowed_tools:
5
+ - mcp__e2e-runner__e2e_pool_status
6
+ - mcp__e2e-runner__e2e_list
7
+ - mcp__e2e-runner__e2e_run
8
+ - mcp__e2e-runner__e2e_screenshot
9
+ - mcp__e2e-runner__e2e_network_logs
10
+ - mcp__e2e-runner__e2e_learnings
11
+ ---
12
+
13
+ # Run E2E Tests
14
+
15
+ Execute E2E tests and provide a complete analysis of results.
16
+
17
+ ## Workflow
18
+
19
+ 1. **Check pool availability** — Call `e2e_pool_status` to confirm the Chrome pool is running. If not available, tell the user to run `npx e2e-runner pool start` via CLI.
20
+
21
+ 2. **List available suites** — Call `e2e_list` to show the user what test suites are available.
22
+
23
+ 3. **Run tests** — Call `e2e_run` based on user input:
24
+ - If user specified a suite name: use `suite` parameter
25
+ - If user specified a file: use `file` parameter
26
+ - If user said "all" or didn't specify: use `all: true`
27
+ - Always pass `cwd` with the current working directory
28
+ - Pass any user-specified overrides: `baseUrl`, `concurrency`, `retries`, `failOnNetworkError`
29
+
30
+ 4. **Analyze results** — Parse the run response:
31
+ - Report pass/fail summary and duration
32
+ - For failures: show error messages and retrieve error screenshots with `e2e_screenshot`
33
+ - For verifications (tests with `expect`): retrieve verification screenshots and judge against descriptions
34
+ - Highlight flaky tests if any
35
+ - Summarize network activity (failed requests, slow requests)
36
+
37
+ 5. **Drill down if needed** — For failed tests:
38
+ - Use `e2e_network_logs` with `runDbId` to investigate network failures
39
+ - Use `e2e_learnings` to check if this is a known pattern or new failure
40
+
41
+ 6. **Report** — Provide a clear summary to the user with actionable next steps.
42
+
43
+ ## Arguments
44
+
45
+ The user may pass arguments after the command:
46
+ - Suite name: `/e2e-runner:run auth` → run the auth suite
47
+ - `--all`: run all suites
48
+ - `--base-url <url>`: override base URL
49
+ - `--retries <n>`: set retry count
@@ -0,0 +1,63 @@
1
+ ---
2
+ description: Verify a GitHub/GitLab issue by creating and running E2E tests
3
+ user_invocable: true
4
+ allowed_tools:
5
+ - mcp__e2e-runner__e2e_pool_status
6
+ - mcp__e2e-runner__e2e_issue
7
+ - mcp__e2e-runner__e2e_create_test
8
+ - mcp__e2e-runner__e2e_run
9
+ - mcp__e2e-runner__e2e_screenshot
10
+ - mcp__e2e-runner__e2e_network_logs
11
+ - mcp__e2e-runner__e2e_capture
12
+ - Read
13
+ - Grep
14
+ ---
15
+
16
+ # Verify Issue
17
+
18
+ Turn a GitHub or GitLab bug report into executable E2E tests to confirm or dismiss the bug.
19
+
20
+ ## Workflow
21
+
22
+ 1. **Check pool** — Call `e2e_pool_status` to ensure the Chrome pool is available.
23
+
24
+ 2. **Fetch the issue** — Call `e2e_issue` with the issue URL. Default `mode: "prompt"` returns issue details + a structured prompt for test creation.
25
+
26
+ 3. **Analyze the issue** — Parse the issue details:
27
+ - Understand the reported bug or expected behavior
28
+ - Identify affected pages/flows
29
+ - Note any reproduction steps provided
30
+
31
+ 4. **Explore the app** — Use `e2e_capture` to screenshot relevant pages. Use `Read` and `Grep` to check source code for related components, API endpoints, or selectors.
32
+
33
+ 5. **Design tests** — Create tests that assert the **correct behavior**:
34
+ - If tests **fail** → bug is confirmed (correct behavior is not working)
35
+ - If tests **pass** → bug is not reproducible
36
+
37
+ 6. **Create and run** — Use `e2e_create_test` to write the test file, then `e2e_run` to execute it.
38
+
39
+ 7. **Analyze results** — For failures:
40
+ - Retrieve error screenshots with `e2e_screenshot`
41
+ - Check network logs with `e2e_network_logs` for API-related issues
42
+ - Determine if the failure confirms the bug
43
+
44
+ 8. **Report verdict** — Clearly state:
45
+ - **BUG CONFIRMED**: tests failed, reproducing the issue
46
+ - **NOT REPRODUCIBLE**: tests passed, correct behavior works as expected
47
+ - Include evidence (screenshots, error messages, network details)
48
+
49
+ ## Alternative: Verify Mode
50
+
51
+ If `ANTHROPIC_API_KEY` is set, use `e2e_issue` with `mode: "verify"` for a fully automated flow — it generates tests via Claude API, runs them, and reports the result.
52
+
53
+ ## Arguments
54
+
55
+ **Required**: GitHub or GitLab issue URL
56
+
57
+ ```
58
+ /e2e-runner:verify-issue https://github.com/org/repo/issues/123
59
+ ```
60
+
61
+ Optional flags:
62
+ - `--test-type api` — generate API tests instead of UI tests
63
+ - `--verify` — use verify mode (requires ANTHROPIC_API_KEY)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@matware/e2e-runner",
3
- "version": "1.1.1",
3
+ "version": "1.2.1",
4
4
  "mcpName": "io.github.fastslack/e2e-runner",
5
5
  "description": "E2E test runner using Chrome Pool (browserless/chrome) with parallel execution",
6
6
  "type": "module",
@@ -15,7 +15,12 @@
15
15
  "files": [
16
16
  "bin/",
17
17
  "src/",
18
- "templates/"
18
+ "templates/",
19
+ ".claude-plugin/",
20
+ ".mcp.json",
21
+ "skills/",
22
+ "commands/",
23
+ "agents/"
19
24
  ],
20
25
  "keywords": [
21
26
  "e2e",
@@ -41,6 +46,9 @@
41
46
  "better-sqlite3": "^11.0.0",
42
47
  "puppeteer-core": "^24.0.0"
43
48
  },
49
+ "scripts": {
50
+ "build:dashboard": "node templates/build-dashboard.js"
51
+ },
44
52
  "engines": {
45
53
  "node": ">=20.0.0"
46
54
  }
@@ -0,0 +1,166 @@
1
+ ---
2
+ name: e2e-testing
3
+ description: Create, run, and debug JSON-driven E2E browser tests with Chrome pool integration
4
+ ---
5
+
6
+ # E2E Testing with @matware/e2e-runner
7
+
8
+ ## Overview
9
+
10
+ `@matware/e2e-runner` is a JSON-driven E2E test runner. Tests are defined as JSON files with sequential browser actions — no JavaScript test code. Tests run in parallel against a Chrome pool (browserless/chrome via Docker) using Puppeteer.
11
+
12
+ **Key capabilities:** 13 MCP tools for running tests, creating test files, capturing screenshots, analyzing network traffic, verifying GitHub/GitLab issues, and querying a learning system for stability insights.
13
+
14
+ ## Prerequisites
15
+
16
+ Before running any tests, verify the Chrome pool is available:
17
+
18
+ ```
19
+ e2e_pool_status → check "Available: yes" and session capacity
20
+ ```
21
+
22
+ If the pool is not running, the user must start it via CLI (not available via MCP):
23
+ ```bash
24
+ npx e2e-runner pool start
25
+ ```
26
+
27
+ ## Core Workflow
28
+
29
+ The standard test execution flow:
30
+
31
+ 1. **Check pool** → `e2e_pool_status` — confirm Chrome pool is ready
32
+ 2. **List suites** → `e2e_list` — discover available test files and modules
33
+ 3. **Run tests** → `e2e_run` — execute with `all`, `suite`, or `file` parameter
34
+ 4. **Interpret results** — check `summary`, `failures`, `narratives`, `networkSummary`
35
+ 5. **View screenshots** → `e2e_screenshot` — retrieve error/verification screenshots by `ss:HASH`
36
+ 6. **Drill into network** → `e2e_network_logs` — use `runDbId` to inspect requests/responses
37
+ 7. **Check learnings** → `e2e_learnings` — query stability trends, flaky tests, error patterns
38
+
39
+ ### Interpreting Run Results
40
+
41
+ The `e2e_run` response includes:
42
+
43
+ - **summary**: pass/fail counts, duration, `runDbId` for drill-down
44
+ - **failures**: failed test names with error messages and error screenshot hashes
45
+ - **narratives**: step-by-step human-readable story of each test execution
46
+ - **networkSummary**: per-test request stats (status distribution, slow/failed requests)
47
+ - **verifications**: tests with `expect` field — call `e2e_screenshot` to visually verify
48
+ - **learnings**: stability insights from the learning system (new failures, flaky patterns)
49
+
50
+ ## Creating Tests
51
+
52
+ ### Basic Structure
53
+
54
+ ```json
55
+ [
56
+ {
57
+ "name": "login-flow",
58
+ "actions": [
59
+ { "type": "goto", "value": "/login" },
60
+ { "type": "type", "selector": "#email", "value": "user@example.com" },
61
+ { "type": "type", "selector": "#password", "value": "secret" },
62
+ { "type": "click", "text": "Sign In" },
63
+ { "type": "wait", "selector": ".dashboard" },
64
+ { "type": "assert_url", "value": "/dashboard" }
65
+ ]
66
+ }
67
+ ]
68
+ ```
69
+
70
+ Use `e2e_create_test` to write test files. Use `e2e_create_module` for reusable action sequences.
71
+
72
+ ### Key Action Patterns
73
+
74
+ - **Navigation**: `goto` (full page load), `navigate` (SPA-friendly, non-blocking)
75
+ - **Interaction**: `click` (selector or text), `type`/`fill`, `select`, `press`, `hover`, `scroll`
76
+ - **React/MUI**: `type_react` (controlled inputs), `click_option`, `focus_autocomplete`, `click_chip`, `click_regex`
77
+ - **Assertions**: `assert_text` (page-wide), `assert_element_text` (scoped), `assert_url`, `assert_visible`, `assert_not_visible`, `assert_count`, `assert_attribute`, `assert_class`, `assert_input_value`, `assert_matches`
78
+ - **Extraction**: `get_text` (non-assertion, returns element text), `screenshot`
79
+ - **Advanced**: `evaluate` (run JS in browser), `assert_no_network_errors`, `clear_cookies`
80
+
81
+ ### Visual Verification
82
+
83
+ Add an `expect` field to any test for AI-powered visual verification:
84
+
85
+ ```json
86
+ {
87
+ "name": "dashboard-loads",
88
+ "expect": "Should show patient list with at least 3 rows and no error messages",
89
+ "actions": [...]
90
+ }
91
+ ```
92
+
93
+ After running, call `e2e_screenshot` with each verification hash and judge the screenshot against the description.
94
+
95
+ ### Reusable Modules
96
+
97
+ Create modules with `e2e_create_module`, reference them in tests:
98
+
99
+ ```json
100
+ { "$use": "auth-jwt", "params": { "email": "admin@test.com" } }
101
+ ```
102
+
103
+ For complete action type reference, see [action-types.md](references/action-types.md).
104
+ For JSON format details (hooks, serial, retries, modules), see [test-json-format.md](references/test-json-format.md).
105
+
106
+ ## Issue Verification
107
+
108
+ Turn GitHub/GitLab bug reports into executable tests:
109
+
110
+ ### Prompt Mode (default, no API key needed)
111
+
112
+ 1. `e2e_issue` with issue URL → returns structured prompt with issue details
113
+ 2. Analyze the issue and design test actions
114
+ 3. `e2e_create_test` → create the test file
115
+ 4. `e2e_run` → execute and verify
116
+
117
+ ### Verify Mode (requires ANTHROPIC_API_KEY)
118
+
119
+ 1. `e2e_issue` with `mode: "verify"` → auto-generates tests via Claude API, runs them, reports result
120
+ 2. Test failure = bug confirmed, all pass = not reproducible
121
+
122
+ Supports both UI tests (`testType: "e2e"`) and API tests (`testType: "api"`).
123
+
124
+ ## Debugging & Analysis
125
+
126
+ ### Network Inspection
127
+
128
+ ```
129
+ e2e_network_logs(runDbId) → all requests
130
+ e2e_network_logs(runDbId, errorsOnly: true) → failed requests only
131
+ e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
132
+ e2e_network_logs(runDbId, urlPattern: "/api/") → filter by URL pattern
133
+ ```
134
+
135
+ ### Learning System
136
+
137
+ ```
138
+ e2e_learnings("summary") → full project overview
139
+ e2e_learnings("flaky") → flaky test analysis
140
+ e2e_learnings("selectors") → selector stability
141
+ e2e_learnings("errors") → recurring error patterns
142
+ e2e_learnings("test:name") → drill into specific test history
143
+ ```
144
+
145
+ ### On-Demand Capture
146
+
147
+ Use `e2e_capture` to screenshot any URL without running a full test suite. Useful for visual exploration or verifying current state.
148
+
149
+ ### Dashboard
150
+
151
+ Start/stop the web dashboard with `e2e_dashboard_start` / `e2e_dashboard_stop` for a visual UI at `http://localhost:8484`.
152
+
153
+ ## Important Rules
154
+
155
+ 1. **Always pass `cwd`** — All MCP tools accept `cwd` (the project root). Always pass it so config files and test directories resolve correctly.
156
+ 2. **`baseUrl` default is `http://host.docker.internal:3000`** — Chrome runs inside Docker, so it uses `host.docker.internal` to reach the host machine. Override with `baseUrl` if the app runs on a different port.
157
+ 3. **Pool management is CLI-only** — `pool start` and `pool stop` are not available via MCP. Only `e2e_pool_status` is an MCP tool.
158
+ 4. **`evaluate` is strict** — Returns starting with `FAIL:`/`ERROR:` or returning `false` will fail the test. Prefer granular assertion actions over `evaluate` with inline JS.
159
+ 5. **Serial tests** — Mark tests with `"serial": true` if they share mutable state. They run after all parallel tests.
160
+ 6. **Action retries** — Use `"retries": N` on individual actions for flaky selectors, or globally via config.
161
+
162
+ ## References
163
+
164
+ - [Action Types Reference](references/action-types.md) — Complete catalog of 28+ action types with fields and examples
165
+ - [Test JSON Format](references/test-json-format.md) — JSON structure, hooks, serial, retries, modules, exclude patterns, environment profiles
166
+ - [Troubleshooting](references/troubleshooting.md) — Common problems and solutions
@@ -0,0 +1,100 @@
1
+ # Action Types Reference
2
+
3
+ Complete catalog of all action types supported by @matware/e2e-runner.
4
+
5
+ ## Navigation
6
+
7
+ | Action | Fields | Description |
8
+ |--------|--------|-------------|
9
+ | `goto` | `value` (URL or path) | Full page navigation. Relative paths are prefixed with `baseUrl`. Waits for `domcontentloaded`. |
10
+ | `navigate` | `value` (URL or path) | SPA-friendly navigation. Uses `load` event with a 5s race timeout — won't block if client-side routing doesn't fire `load`. |
11
+
12
+ ## Interaction
13
+
14
+ | Action | Fields | Description |
15
+ |--------|--------|-------------|
16
+ | `click` | `selector` OR `text` | Click by CSS selector or by visible text content. Text search covers: `button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"], [role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6, dd, dt`. |
17
+ | `type` / `fill` | `selector`, `value` | Triple-clicks to select all, then Backspace to clear, then types with 20ms delay per character. |
18
+ | `select` | `selector`, `value` | Select an `<option>` value in a `<select>` element. |
19
+ | `clear` | `selector` | Triple-click + Backspace to clear an input field. |
20
+ | `press` | `value` (key name) | Press a keyboard key (e.g. `"Enter"`, `"Tab"`, `"Escape"`, `"ArrowDown"`). |
21
+ | `scroll` | `selector` (optional), `value` (optional) | Scroll element into view, or scroll window by Y pixels (default 300). |
22
+ | `hover` | `selector` | Hover over an element. |
23
+
24
+ ## Framework-Aware (React/MUI)
25
+
26
+ | Action | Fields | Description |
27
+ |--------|--------|-------------|
28
+ | `type_react` | `selector`, `value` | Types into React controlled inputs using native value setter. Dispatches `input` + `change` events so React state updates. Supports `<input>` and `<textarea>`. |
29
+ | `click_regex` | `text` (regex), `selector` (optional), `value` (`"last"` optional) | Click element whose textContent matches regex (case-insensitive). Default: first match. `value: "last"` for last match. `selector` scopes the search. |
30
+ | `click_option` | `text` | Click a `[role="option"]` element by text — for autocomplete/select dropdowns. Waits for option to appear. |
31
+ | `focus_autocomplete` | `text` (label text) | Focus an autocomplete input by label. Supports MUI `.MuiAutocomplete-root` and `[role="combobox"]`. |
32
+ | `click_chip` | `text` | Click a chip/tag element by text. Searches `[class*="Chip"]`, `[class*="chip"]`, `[data-chip]`. |
33
+
34
+ ## Assertions
35
+
36
+ | Action | Fields | Description |
37
+ |--------|--------|-------------|
38
+ | `assert_text` | `text` | Check entire page body contains text (substring match). |
39
+ | `assert_element_text` | `selector`, `text`, `value` (`"exact"` optional) | Check specific element's `textContent`. Default: substring match. With `value: "exact"`: strict `trim() ===` comparison. |
40
+ | `assert_url` | `value` | Check current URL. Path-only (`/dashboard`) compares pathname. Full URL does substring match. |
41
+ | `assert_visible` | `selector` | Element exists and is visible (`display`, `visibility`, `opacity` checks). |
42
+ | `assert_not_visible` | `selector` | Passes if element doesn't exist OR is hidden. |
43
+ | `assert_count` | `selector`, `value` | Count matching elements. Supports exact (`"5"`) and operators (`">3"`, `">=1"`, `"<10"`, `"<=5"`). |
44
+ | `assert_attribute` | `selector`, `value` (`"attr=expected"` or `"attr"`) | With `=`: checks attribute value. Without: checks attribute existence. |
45
+ | `assert_class` | `selector`, `value` | Checks `classList.contains(value)`. |
46
+ | `assert_input_value` | `selector`, `value` | Checks `element.value.includes(value)` on input/select/textarea. |
47
+ | `assert_matches` | `selector`, `value` (regex) | Tests element's `textContent` against `new RegExp(value)`. |
48
+ | `assert_no_network_errors` | — | Checks accumulated `requestfailed` events during the test. Fails with error details if any exist. |
49
+
50
+ ### Assertion Disambiguation
51
+
52
+ - **`assert_text`** → searches the **entire page body** (substring)
53
+ - **`assert_element_text`** → checks a **specific element** (substring, or exact with `value: "exact"`)
54
+ - **`assert_matches`** → checks a specific element against a **regex** pattern
55
+ - **`assert_input_value`** → reads the `.value` property (for form fields)
56
+
57
+ ## Extraction & Utility
58
+
59
+ | Action | Fields | Description |
60
+ |--------|--------|-------------|
61
+ | `get_text` | `selector` | Returns `{ value: textContent.trim() }`. Non-assertion — never fails. |
62
+ | `screenshot` | `value` (filename, optional) | Captures screenshot. Filename gets timestamp suffix for uniqueness. |
63
+ | `wait` | `selector` OR `text` OR `value` (ms) | Wait for selector, text on page, or fixed delay. |
64
+ | `evaluate` | `value` (JS code) | Run JavaScript in browser context. **Strict**: returns starting with `FAIL:`/`ERROR:` → test fails. Returns `false` → test fails. |
65
+ | `clear_cookies` | `value` (origin, optional) | Clears cookies, localStorage, sessionStorage for origin. |
66
+
67
+ ## Action-Level Retry
68
+
69
+ Any action can have `"retries": N` for per-action retry on failure:
70
+
71
+ ```json
72
+ { "type": "click", "selector": "#dynamic-btn", "retries": 3 }
73
+ { "type": "wait", "selector": ".lazy-loaded", "retries": 2 }
74
+ ```
75
+
76
+ Delay between retries: `actionRetryDelay` config (default 500ms).
77
+
78
+ ## Examples
79
+
80
+ ### React input + autocomplete flow
81
+ ```json
82
+ { "type": "focus_autocomplete", "text": "Diagnosis" },
83
+ { "type": "type_react", "selector": "#diagnosis-input", "value": "Cefalea" },
84
+ { "type": "click_option", "text": "Cefalea tensional" }
85
+ ```
86
+
87
+ ### Regex click (last match)
88
+ ```json
89
+ { "type": "click_regex", "text": "start encounter", "selector": "button", "value": "last" }
90
+ ```
91
+
92
+ ### Form validation assertions
93
+ ```json
94
+ { "type": "assert_attribute", "selector": "input#email", "value": "type=email" },
95
+ { "type": "assert_attribute", "selector": "button.submit", "value": "disabled" },
96
+ { "type": "assert_class", "selector": ".nav-item:first-child", "value": "active" },
97
+ { "type": "assert_input_value", "selector": "#email", "value": "user@example.com" },
98
+ { "type": "assert_matches", "selector": ".phone", "value": "\\d{3}-\\d{3}-\\d{4}" },
99
+ { "type": "assert_count", "selector": ".table-row", "value": ">3" }
100
+ ```
@@ -0,0 +1,159 @@
1
+ # Test JSON Format Reference
2
+
3
+ ## Basic Format (Array)
4
+
5
+ A test file is a JSON array of test objects:
6
+
7
+ ```json
8
+ [
9
+ {
10
+ "name": "test-name",
11
+ "actions": [
12
+ { "type": "goto", "value": "/page" },
13
+ { "type": "assert_text", "text": "Expected content" }
14
+ ]
15
+ }
16
+ ]
17
+ ```
18
+
19
+ ## Object Format (with Hooks)
20
+
21
+ When hooks are needed, use the object format:
22
+
23
+ ```json
24
+ {
25
+ "hooks": {
26
+ "beforeAll": [{ "type": "goto", "value": "/setup" }],
27
+ "beforeEach": [{ "type": "goto", "value": "/" }],
28
+ "afterEach": [],
29
+ "afterAll": []
30
+ },
31
+ "tests": [
32
+ { "name": "test-1", "actions": [...] }
33
+ ]
34
+ }
35
+ ```
36
+
37
+ **Hook lifecycle:**
38
+ - `beforeAll` — runs once before all tests (on a separate browser page, state does NOT carry over)
39
+ - `beforeEach` — runs before each individual test (on the test's own page)
40
+ - `afterEach` — runs after each test
41
+ - `afterAll` — runs once after all tests
42
+
43
+ > **Warning**: `beforeAll` runs on a separate page that closes before tests start. Don't use it for browser state setup (cookies, localStorage). Use `beforeEach` instead.
44
+
45
+ ## Test Options
46
+
47
+ | Field | Type | Description |
48
+ |-------|------|-------------|
49
+ | `name` | string | **Required.** Test identifier. |
50
+ | `actions` | array | **Required.** Sequential browser actions. |
51
+ | `expect` | string | Visual verification description. Triggers auto-screenshot + AI judgment. |
52
+ | `serial` | boolean | Run sequentially after all parallel tests (for shared state). |
53
+ | `retries` | number | Per-test retry count on failure. Overrides global config. |
54
+ | `timeout` | number | Per-test timeout in ms. Overrides global `testTimeout` (default 60000). |
55
+
56
+ ## Serial Tests
57
+
58
+ Tests that share mutable state should be marked serial to prevent race conditions:
59
+
60
+ ```json
61
+ { "name": "create-record", "serial": true, "actions": [...] },
62
+ { "name": "verify-record", "serial": true, "actions": [...] }
63
+ ```
64
+
65
+ Serial tests run one-at-a-time **after** all parallel tests finish.
66
+
67
+ ## Retry Behavior
68
+
69
+ ### Test-level retries
70
+ ```json
71
+ { "name": "flaky-test", "retries": 3, "actions": [...] }
72
+ ```
73
+
74
+ Or globally: `--retries 2` / `retries: 2` in config. Each retry gets its own timeout. Flaky tests (pass after retry) are logged as "flaky".
75
+
76
+ ### Action-level retries
77
+ ```json
78
+ { "type": "click", "selector": "#dynamic-btn", "retries": 3 }
79
+ ```
80
+
81
+ Or globally: `--action-retries 2`. Delay between action retries: `actionRetryDelay` (default 500ms).
82
+
83
+ ## Reusable Modules
84
+
85
+ Create modules with `e2e_create_module`, reference them in tests:
86
+
87
+ ```json
88
+ {
89
+ "name": "login-test",
90
+ "actions": [
91
+ { "$use": "auth-login", "params": { "email": "admin@test.com", "password": "secret" } },
92
+ { "type": "assert_url", "value": "/dashboard" }
93
+ ]
94
+ }
95
+ ```
96
+
97
+ Module definition (in `e2e/modules/auth-login.json`):
98
+ ```json
99
+ {
100
+ "$module": "auth-login",
101
+ "description": "Log in with email/password",
102
+ "params": {
103
+ "email": { "required": true, "description": "User email" },
104
+ "password": { "required": true, "description": "User password" }
105
+ },
106
+ "actions": [
107
+ { "type": "goto", "value": "/login" },
108
+ { "type": "type", "selector": "#email", "value": "{{email}}" },
109
+ { "type": "type", "selector": "#password", "value": "{{password}}" },
110
+ { "type": "click", "text": "Sign In" },
111
+ { "type": "wait", "selector": ".dashboard" }
112
+ ]
113
+ }
114
+ ```
115
+
116
+ ## Suite Naming & Ordering
117
+
118
+ Files can have numeric prefixes for execution order:
119
+ - `01-auth.json`, `02-dashboard.json`, `03-settings.json`
120
+
121
+ The `--suite` flag strips the prefix when matching: `--suite auth` finds `01-auth.json`.
122
+
123
+ ## Excluding Tests
124
+
125
+ Use `exclude` in config to skip files when running `--all`:
126
+
127
+ ```js
128
+ // e2e.config.js
129
+ export default {
130
+ exclude: ['explore-*', 'debug-*', 'draft-*']
131
+ };
132
+ ```
133
+
134
+ Individual `--suite` runs are not affected by exclude patterns.
135
+
136
+ ## Environment Profiles
137
+
138
+ Define named profiles in config:
139
+
140
+ ```js
141
+ // e2e.config.js
142
+ export default {
143
+ baseUrl: 'http://host.docker.internal:3000',
144
+ environments: {
145
+ staging: { baseUrl: 'https://staging.example.com' },
146
+ production: { baseUrl: 'https://example.com', concurrency: 5 }
147
+ }
148
+ };
149
+ ```
150
+
151
+ Activate with `--env staging` or `E2E_ENV=staging`. Profile values override all other config.
152
+
153
+ ## Config Priority (ascending)
154
+
155
+ 1. Hardcoded defaults
156
+ 2. `e2e.config.js` or `e2e.config.json`
157
+ 3. Environment variables (`BASE_URL`, `CONCURRENCY`, etc.)
158
+ 4. CLI flags (`--base-url`, `--concurrency`, etc.)
159
+ 5. Environment profile merge (via `--env`)