@matware/e2e-runner 1.2.1 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +21 -0
- package/.mcp.json +2 -2
- package/.opencode/commands/create-test.md +63 -0
- package/.opencode/commands/run.md +50 -0
- package/.opencode/commands/verify-issue.md +62 -0
- package/.opencode/skills/e2e-testing/SKILL.md +181 -0
- package/.opencode/skills/e2e-testing/references/action-types.md +143 -0
- package/.opencode/skills/e2e-testing/references/auth-strategies.md +91 -0
- package/.opencode/skills/e2e-testing/references/graphql.md +59 -0
- package/.opencode/skills/e2e-testing/references/issue-verification.md +59 -0
- package/.opencode/skills/e2e-testing/references/multi-pool.md +60 -0
- package/.opencode/skills/e2e-testing/references/network-debugging.md +62 -0
- package/.opencode/skills/e2e-testing/references/test-json-format.md +163 -0
- package/.opencode/skills/e2e-testing/references/troubleshooting.md +224 -0
- package/.opencode/skills/e2e-testing/references/variables.md +41 -0
- package/.opencode/skills/e2e-testing/references/visual-verification.md +89 -0
- package/OPENCODE.md +166 -0
- package/README.md +581 -55
- package/agents/test-creator.md +54 -1
- package/agents/test-improver.md +37 -0
- package/bin/cli.js +408 -16
- package/commands/create-test.md +16 -1
- package/opencode.json +11 -0
- package/package.json +7 -2
- package/scripts/setup-opencode.sh +113 -0
- package/skills/e2e-testing/SKILL.md +10 -3
- package/skills/e2e-testing/references/action-types.md +48 -5
- package/skills/e2e-testing/references/auth-strategies.md +91 -0
- package/skills/e2e-testing/references/graphql.md +59 -0
- package/skills/e2e-testing/references/issue-verification.md +59 -0
- package/skills/e2e-testing/references/multi-pool.md +60 -0
- package/skills/e2e-testing/references/network-debugging.md +62 -0
- package/skills/e2e-testing/references/test-json-format.md +4 -0
- package/skills/e2e-testing/references/troubleshooting.md +44 -2
- package/skills/e2e-testing/references/variables.md +41 -0
- package/skills/e2e-testing/references/visual-verification.md +89 -0
- package/src/actions.js +324 -2
- package/src/ai-generate.js +58 -8
- package/src/config.js +143 -0
- package/src/dashboard.js +145 -13
- package/src/db.js +130 -2
- package/src/index.js +7 -6
- package/src/learner-sqlite.js +304 -0
- package/src/learner.js +8 -3
- package/src/mcp-tools.js +1121 -43
- package/src/module-resolver.js +37 -0
- package/src/narrate.js +37 -0
- package/src/pool-manager.js +223 -0
- package/src/reporter.js +82 -1
- package/src/runner.js +157 -28
- package/src/sync/auth.js +354 -0
- package/src/sync/client.js +572 -0
- package/src/sync/hub-routes.js +816 -0
- package/src/sync/index.js +68 -0
- package/src/sync/middleware.js +347 -0
- package/src/sync/queue.js +209 -0
- package/src/sync/schema.js +540 -0
- package/src/verify.js +10 -7
- package/src/watch.js +384 -0
- package/templates/build-dashboard.js +47 -6
- package/templates/dashboard/js/api.js +60 -0
- package/templates/dashboard/js/init.js +13 -0
- package/templates/dashboard/js/keyboard.js +46 -0
- package/templates/dashboard/js/state.js +40 -0
- package/templates/dashboard/js/toast.js +41 -0
- package/templates/dashboard/js/utils.js +196 -0
- package/templates/dashboard/js/view-live.js +143 -0
- package/templates/dashboard/js/view-runs.js +572 -0
- package/templates/dashboard/js/view-tests.js +294 -0
- package/templates/dashboard/js/view-watch.js +242 -0
- package/templates/dashboard/js/websocket.js +110 -0
- package/templates/dashboard/styles/base.css +69 -0
- package/templates/dashboard/styles/components.css +110 -0
- package/templates/dashboard/styles/view-live.css +74 -0
- package/templates/dashboard/styles/view-runs.css +207 -0
- package/templates/dashboard/styles/view-tests.css +96 -0
- package/templates/dashboard/styles/view-watch.css +53 -0
- package/templates/dashboard/template.html +165 -99
- package/templates/dashboard.html +1596 -541
- package/templates/sample-test.json +0 -8
- package/templates/dashboard/app.js +0 -1152
- package/templates/dashboard/styles.css +0 -413
package/commands/create-test.md
CHANGED
|
@@ -37,11 +37,26 @@ Help the user create a new E2E test file by exploring the application and design
|
|
|
37
37
|
- Add `wait` actions before assertions on dynamic content
|
|
38
38
|
- Add `assert_no_network_errors` after critical page loads
|
|
39
39
|
- Consider adding an `expect` field for visual verification
|
|
40
|
+
- **DRY**: If auth/setup is repeated across tests, use `beforeEach` hook (object format) instead of repeating per test
|
|
41
|
+
- **DRY**: If 3+ tests repeat the same action pattern, create a module with `e2e_create_module` first
|
|
40
42
|
|
|
41
|
-
7. **Create the test** — Call `e2e_create_test` with the designed test structure.
|
|
43
|
+
7. **Create the test** — Call `e2e_create_test` with the designed test structure.
|
|
44
|
+
- Check existing modules (`e2e/modules/`) — reuse them via `$use` instead of duplicating actions
|
|
45
|
+
- Create new modules with `e2e_create_module` for repeated sequences (auth, navigation, screenshot patterns)
|
|
46
|
+
- Use object format `{ "beforeEach": [...], "tests": [...] }` when hooks are needed
|
|
42
47
|
|
|
43
48
|
8. **Validate** — Run the newly created test with `e2e_run` using the `suite` parameter. Analyze results and iterate if needed.
|
|
44
49
|
|
|
50
|
+
## Naming Rules (CRITICAL)
|
|
51
|
+
|
|
52
|
+
Suite names MUST be unique and specific to the feature, issue, or user flow:
|
|
53
|
+
- GOOD: `login-valid-credentials`, `issue-1743-auth-redirect`, `checkout-payment-flow`
|
|
54
|
+
- BAD: `all`, `test`, `debug`, `new`, `temp`, `main`, `suite`
|
|
55
|
+
|
|
56
|
+
If testing a GitHub/GitLab issue, include the issue number: `issue-1743-auth-timeout`, `bug-502-duplicate-submit`
|
|
57
|
+
|
|
58
|
+
Before creating, always call `e2e_list` to verify the name doesn't already exist.
|
|
59
|
+
|
|
45
60
|
## Arguments
|
|
46
61
|
|
|
47
62
|
The user may provide:
|
package/opencode.json
ADDED
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@matware/e2e-runner",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.3.0",
|
|
4
4
|
"mcpName": "io.github.fastslack/e2e-runner",
|
|
5
5
|
"description": "E2E test runner using Chrome Pool (browserless/chrome) with parallel execution",
|
|
6
6
|
"type": "module",
|
|
@@ -20,7 +20,11 @@
|
|
|
20
20
|
".mcp.json",
|
|
21
21
|
"skills/",
|
|
22
22
|
"commands/",
|
|
23
|
-
"agents/"
|
|
23
|
+
"agents/",
|
|
24
|
+
".opencode/",
|
|
25
|
+
"opencode.json",
|
|
26
|
+
"OPENCODE.md",
|
|
27
|
+
"scripts/setup-opencode.sh"
|
|
24
28
|
],
|
|
25
29
|
"keywords": [
|
|
26
30
|
"e2e",
|
|
@@ -31,6 +35,7 @@
|
|
|
31
35
|
"parallel",
|
|
32
36
|
"mcp",
|
|
33
37
|
"claude-code",
|
|
38
|
+
"opencode",
|
|
34
39
|
"github-issues",
|
|
35
40
|
"ai-testing"
|
|
36
41
|
],
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# Setup @matware/e2e-runner for OpenCode
|
|
3
|
+
# Usage: ./setup-opencode.sh [--global]
|
|
4
|
+
|
|
5
|
+
set -e
|
|
6
|
+
|
|
7
|
+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
8
|
+
PACKAGE_DIR="$(dirname "$SCRIPT_DIR")"
|
|
9
|
+
|
|
10
|
+
# Colors
|
|
11
|
+
RED='\033[0;31m'
|
|
12
|
+
GREEN='\033[0;32m'
|
|
13
|
+
YELLOW='\033[1;33m'
|
|
14
|
+
NC='\033[0m' # No Color
|
|
15
|
+
|
|
16
|
+
log_info() { echo -e "${GREEN}✓${NC} $1"; }
|
|
17
|
+
log_warn() { echo -e "${YELLOW}!${NC} $1"; }
|
|
18
|
+
log_error() { echo -e "${RED}✗${NC} $1"; }
|
|
19
|
+
|
|
20
|
+
# Check if --global flag is passed
|
|
21
|
+
GLOBAL_INSTALL=false
|
|
22
|
+
if [[ "$1" == "--global" ]]; then
|
|
23
|
+
GLOBAL_INSTALL=true
|
|
24
|
+
fi
|
|
25
|
+
|
|
26
|
+
if [ "$GLOBAL_INSTALL" = true ]; then
|
|
27
|
+
echo "Setting up e2e-runner for OpenCode (global)..."
|
|
28
|
+
|
|
29
|
+
# Global config directory
|
|
30
|
+
OPENCODE_CONFIG="${HOME}/.config/opencode"
|
|
31
|
+
|
|
32
|
+
# Create directories
|
|
33
|
+
mkdir -p "${OPENCODE_CONFIG}/skills"
|
|
34
|
+
mkdir -p "${OPENCODE_CONFIG}/commands"
|
|
35
|
+
|
|
36
|
+
# Copy skills
|
|
37
|
+
if [ -d "${PACKAGE_DIR}/.opencode/skills" ]; then
|
|
38
|
+
cp -r "${PACKAGE_DIR}/.opencode/skills/"* "${OPENCODE_CONFIG}/skills/"
|
|
39
|
+
log_info "Installed skills to ${OPENCODE_CONFIG}/skills/"
|
|
40
|
+
fi
|
|
41
|
+
|
|
42
|
+
# Copy commands
|
|
43
|
+
if [ -d "${PACKAGE_DIR}/.opencode/commands" ]; then
|
|
44
|
+
cp -r "${PACKAGE_DIR}/.opencode/commands/"* "${OPENCODE_CONFIG}/commands/"
|
|
45
|
+
log_info "Installed commands to ${OPENCODE_CONFIG}/commands/"
|
|
46
|
+
fi
|
|
47
|
+
|
|
48
|
+
echo ""
|
|
49
|
+
log_warn "To use the MCP server, add this to your project's opencode.json:"
|
|
50
|
+
echo ""
|
|
51
|
+
cat << 'EOF'
|
|
52
|
+
{
|
|
53
|
+
"mcp": {
|
|
54
|
+
"e2e-runner": {
|
|
55
|
+
"type": "local",
|
|
56
|
+
"command": "e2e-runner-mcp"
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
EOF
|
|
61
|
+
echo ""
|
|
62
|
+
|
|
63
|
+
else
|
|
64
|
+
echo "Setting up e2e-runner for OpenCode (project-local)..."
|
|
65
|
+
|
|
66
|
+
# Check if we're in the package directory or a project that installed it
|
|
67
|
+
if [ -f "${PWD}/opencode.json" ] && [ -d "${PWD}/.opencode" ]; then
|
|
68
|
+
# We're in the package directory
|
|
69
|
+
log_info "Already in e2e-runner package directory"
|
|
70
|
+
exit 0
|
|
71
|
+
fi
|
|
72
|
+
|
|
73
|
+
# Look for node_modules/@matware/e2e-runner
|
|
74
|
+
if [ -d "${PWD}/node_modules/@matware/e2e-runner" ]; then
|
|
75
|
+
PACKAGE_DIR="${PWD}/node_modules/@matware/e2e-runner"
|
|
76
|
+
elif [ ! -f "${PACKAGE_DIR}/package.json" ]; then
|
|
77
|
+
log_error "Cannot find @matware/e2e-runner package"
|
|
78
|
+
log_error "Run: npm install @matware/e2e-runner"
|
|
79
|
+
exit 1
|
|
80
|
+
fi
|
|
81
|
+
|
|
82
|
+
# Copy opencode.json if it doesn't exist
|
|
83
|
+
if [ ! -f "${PWD}/opencode.json" ]; then
|
|
84
|
+
cp "${PACKAGE_DIR}/opencode.json" "${PWD}/opencode.json"
|
|
85
|
+
log_info "Created opencode.json"
|
|
86
|
+
else
|
|
87
|
+
log_warn "opencode.json already exists - merge manually if needed"
|
|
88
|
+
fi
|
|
89
|
+
|
|
90
|
+
# Copy .opencode directory
|
|
91
|
+
mkdir -p "${PWD}/.opencode"
|
|
92
|
+
|
|
93
|
+
if [ -d "${PACKAGE_DIR}/.opencode/skills" ]; then
|
|
94
|
+
mkdir -p "${PWD}/.opencode/skills"
|
|
95
|
+
cp -r "${PACKAGE_DIR}/.opencode/skills/"* "${PWD}/.opencode/skills/"
|
|
96
|
+
log_info "Installed skills to .opencode/skills/"
|
|
97
|
+
fi
|
|
98
|
+
|
|
99
|
+
if [ -d "${PACKAGE_DIR}/.opencode/commands" ]; then
|
|
100
|
+
mkdir -p "${PWD}/.opencode/commands"
|
|
101
|
+
cp -r "${PACKAGE_DIR}/.opencode/commands/"* "${PWD}/.opencode/commands/"
|
|
102
|
+
log_info "Installed commands to .opencode/commands/"
|
|
103
|
+
fi
|
|
104
|
+
fi
|
|
105
|
+
|
|
106
|
+
echo ""
|
|
107
|
+
log_info "OpenCode setup complete!"
|
|
108
|
+
echo ""
|
|
109
|
+
echo "Next steps:"
|
|
110
|
+
echo " 1. Start the Chrome pool: npx e2e-runner pool start"
|
|
111
|
+
echo " 2. Restart OpenCode to load the MCP server"
|
|
112
|
+
echo " 3. Ask: 'Run all E2E tests'"
|
|
113
|
+
echo ""
|
|
@@ -85,7 +85,7 @@ Add an `expect` field to any test for AI-powered visual verification:
|
|
|
85
85
|
```json
|
|
86
86
|
{
|
|
87
87
|
"name": "dashboard-loads",
|
|
88
|
-
"expect": "Should show
|
|
88
|
+
"expect": "Should show data table with at least 3 rows and no error messages",
|
|
89
89
|
"actions": [...]
|
|
90
90
|
}
|
|
91
91
|
```
|
|
@@ -162,5 +162,12 @@ Start/stop the web dashboard with `e2e_dashboard_start` / `e2e_dashboard_stop` f
|
|
|
162
162
|
## References
|
|
163
163
|
|
|
164
164
|
- [Action Types Reference](references/action-types.md) — Complete catalog of 28+ action types with fields and examples
|
|
165
|
-
- [Test JSON Format](references/test-json-format.md) — JSON structure, hooks, serial, retries, modules, exclude patterns, environment profiles
|
|
166
|
-
- [
|
|
165
|
+
- [Test JSON Format](references/test-json-format.md) — JSON structure, hooks, serial, retries, modules, exclude patterns, environment profiles, CI output
|
|
166
|
+
- [GraphQL Action](references/graphql.md) — GQL action config, variables, inline assertions, __e2eGql helper
|
|
167
|
+
- [Authentication Strategies](references/auth-strategies.md) — 6 auth methods + auto-login + reusable auth modules
|
|
168
|
+
- [Network Debugging](references/network-debugging.md) — Error handling, request logging, drill-down pattern
|
|
169
|
+
- [Visual Verification](references/visual-verification.md) — Expect field, double screenshots, strictness levels, verdict format
|
|
170
|
+
- [Multi-Pool Support](references/multi-pool.md) — Config, selection algorithm, failover, pool-aware queue
|
|
171
|
+
- [Variables](references/variables.md) — SQLite-backed variables, syntax, MCP tool, dashboard UI, REST API
|
|
172
|
+
- [Issue Verification](references/issue-verification.md) — GitHub/GitLab, AI modes, test categories, GitLab limitations
|
|
173
|
+
- [Troubleshooting](references/troubleshooting.md) — Common problems, pre-validation, screenshot hashes, dashboard
|
|
@@ -31,6 +31,21 @@ Complete catalog of all action types supported by @matware/e2e-runner.
|
|
|
31
31
|
| `focus_autocomplete` | `text` (label text) | Focus an autocomplete input by label. Supports MUI `.MuiAutocomplete-root` and `[role="combobox"]`. |
|
|
32
32
|
| `click_chip` | `text` | Click a chip/tag element by text. Searches `[class*="Chip"]`, `[class*="chip"]`, `[data-chip]`. |
|
|
33
33
|
|
|
34
|
+
## Storage
|
|
35
|
+
|
|
36
|
+
| Action | Fields | Description |
|
|
37
|
+
|--------|--------|-------------|
|
|
38
|
+
| `set_storage` | `value` (`"key=val"`), `selector` (`"session"` optional) | Set a `localStorage` key (default) or `sessionStorage` key (with `selector: "session"`). |
|
|
39
|
+
| `assert_storage` | `value` (`"key"` or `"key=expected"`), `selector` (`"session"` optional) | Without `=`: checks key exists. With `=`: checks exact value match. Uses `localStorage` by default, `sessionStorage` with `selector: "session"`. |
|
|
40
|
+
|
|
41
|
+
## Smart Interaction
|
|
42
|
+
|
|
43
|
+
| Action | Fields | Description |
|
|
44
|
+
|--------|--------|-------------|
|
|
45
|
+
| `click_icon` | `value` (icon identifier), `selector` (scope, optional) | Click an icon by `data-testid`, `data-icon`, `aria-label`, CSS class, or SVG title. Walks up to nearest clickable ancestor (`button`, `a`, `[role="button"]`). Works with MUI, FontAwesome, Heroicons, Bootstrap Icons, Lucide. |
|
|
46
|
+
| `click_menu_item` | `text` (menu item text), `selector` (scope, optional) | Click a menu item by text. Searches `[role="menuitem"]`, `[role="menuitemradio"]`, `[role="menuitemcheckbox"]`, `.dropdown-item`, `.menu-item`, `[class*="MenuItem"]`, `[role="menu"] > li`. Waits for element to appear. |
|
|
47
|
+
| `click_in_context` | `text` (container text), `selector` (child to click) | Find the smallest container whose text includes `text`, then click the `selector` child within it. Containers: `section`, `article`, `[class*="card"]`, `li`, `tr`, `div[class]`, etc. Both fields required. |
|
|
48
|
+
|
|
34
49
|
## Assertions
|
|
35
50
|
|
|
36
51
|
| Action | Fields | Description |
|
|
@@ -61,9 +76,21 @@ Complete catalog of all action types supported by @matware/e2e-runner.
|
|
|
61
76
|
| `get_text` | `selector` | Returns `{ value: textContent.trim() }`. Non-assertion — never fails. |
|
|
62
77
|
| `screenshot` | `value` (filename, optional) | Captures screenshot. Filename gets timestamp suffix for uniqueness. |
|
|
63
78
|
| `wait` | `selector` OR `text` OR `value` (ms) | Wait for selector, text on page, or fixed delay. |
|
|
64
|
-
| `
|
|
79
|
+
| `wait_network_idle` | `value` (idle ms, default 500), `timeout` (max wait ms, default 30000) | Waits for all network requests to complete. Uses Puppeteer's `page.waitForNetworkIdle()`. Useful after SPA page transitions or data loading. |
|
|
80
|
+
| `evaluate` | `value` (JS code) | Run JavaScript in browser context. See **Strict Evaluate** below. |
|
|
65
81
|
| `clear_cookies` | `value` (origin, optional) | Clears cookies, localStorage, sessionStorage for origin. |
|
|
66
82
|
|
|
83
|
+
### Strict Evaluate Semantics
|
|
84
|
+
|
|
85
|
+
The `evaluate` action checks the return value:
|
|
86
|
+
|
|
87
|
+
- If the JS returns a string starting with `FAIL:`, `ERROR:`, or `FAILED:` → the test **fails** with that message.
|
|
88
|
+
- If the JS returns `false` → the test **fails** (`evaluate returned false`).
|
|
89
|
+
- If the JS returns any other non-null value → stored as `{ value: result }` for visibility.
|
|
90
|
+
- If the JS throws → the test **fails** (standard Puppeteer error).
|
|
91
|
+
|
|
92
|
+
This prevents false PASSes where evaluate actions return error strings that were previously silently ignored.
|
|
93
|
+
|
|
67
94
|
## Action-Level Retry
|
|
68
95
|
|
|
69
96
|
Any action can have `"retries": N` for per-action retry on failure:
|
|
@@ -79,14 +106,14 @@ Delay between retries: `actionRetryDelay` config (default 500ms).
|
|
|
79
106
|
|
|
80
107
|
### React input + autocomplete flow
|
|
81
108
|
```json
|
|
82
|
-
{ "type": "focus_autocomplete", "text": "
|
|
83
|
-
{ "type": "type_react", "selector": "#
|
|
84
|
-
{ "type": "click_option", "text": "
|
|
109
|
+
{ "type": "focus_autocomplete", "text": "Category" },
|
|
110
|
+
{ "type": "type_react", "selector": "#category-input", "value": "Electr" },
|
|
111
|
+
{ "type": "click_option", "text": "Electronics" }
|
|
85
112
|
```
|
|
86
113
|
|
|
87
114
|
### Regex click (last match)
|
|
88
115
|
```json
|
|
89
|
-
{ "type": "click_regex", "text": "
|
|
116
|
+
{ "type": "click_regex", "text": "add to cart", "selector": "button", "value": "last" }
|
|
90
117
|
```
|
|
91
118
|
|
|
92
119
|
### Form validation assertions
|
|
@@ -98,3 +125,19 @@ Delay between retries: `actionRetryDelay` config (default 500ms).
|
|
|
98
125
|
{ "type": "assert_matches", "selector": ".phone", "value": "\\d{3}-\\d{3}-\\d{4}" },
|
|
99
126
|
{ "type": "assert_count", "selector": ".table-row", "value": ">3" }
|
|
100
127
|
```
|
|
128
|
+
|
|
129
|
+
### Storage operations
|
|
130
|
+
```json
|
|
131
|
+
{ "type": "set_storage", "value": "authToken=eyJhbGciOiJIUzI1NiJ9..." },
|
|
132
|
+
{ "type": "assert_storage", "value": "authToken" },
|
|
133
|
+
{ "type": "set_storage", "value": "theme=dark", "selector": "session" },
|
|
134
|
+
{ "type": "assert_storage", "value": "theme=dark", "selector": "session" }
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
### Icon, menu, and contextual clicks
|
|
138
|
+
```json
|
|
139
|
+
{ "type": "click_icon", "value": "edit" },
|
|
140
|
+
{ "type": "click_icon", "value": "delete", "selector": ".user-card" },
|
|
141
|
+
{ "type": "click_menu_item", "text": "Export as PDF" },
|
|
142
|
+
{ "type": "click_in_context", "text": "John Doe", "selector": "button.edit" }
|
|
143
|
+
```
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Authentication Strategies
|
|
2
|
+
|
|
3
|
+
Tests can authenticate using multiple strategies depending on the app's auth mechanism.
|
|
4
|
+
|
|
5
|
+
## 1. UI Login (universal)
|
|
6
|
+
|
|
7
|
+
Fill the login form in `beforeEach`. Works with any auth system — the browser stores cookies/tokens automatically.
|
|
8
|
+
|
|
9
|
+
```json
|
|
10
|
+
{ "hooks": { "beforeEach": [
|
|
11
|
+
{ "type": "goto", "value": "/login" },
|
|
12
|
+
{ "type": "type", "selector": "#email", "value": "test@example.com" },
|
|
13
|
+
{ "type": "type", "selector": "#password", "value": "secret" },
|
|
14
|
+
{ "type": "click", "text": "Sign In" },
|
|
15
|
+
{ "type": "wait", "selector": ".dashboard" }
|
|
16
|
+
]}}
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## 2. JWT Injection (SPAs)
|
|
20
|
+
|
|
21
|
+
Skip the login form by injecting a token into `localStorage` or `sessionStorage`:
|
|
22
|
+
|
|
23
|
+
```json
|
|
24
|
+
{ "type": "set_storage", "value": "accessToken=eyJhbG..." }
|
|
25
|
+
{ "type": "set_storage", "value": "token=eyJhbG...", "selector": "session" }
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## 3. Config-Level Token
|
|
29
|
+
|
|
30
|
+
Set `authToken` + `authStorageKey` in config, env vars (`AUTH_TOKEN`, `AUTH_STORAGE_KEY`), or CLI flags (`--auth-token`, `--auth-storage-key`). Used by `e2e_capture` and `e2e_issue --verify`.
|
|
31
|
+
|
|
32
|
+
## 4. Cookie-Based (server-rendered)
|
|
33
|
+
|
|
34
|
+
Set cookies via `evaluate`:
|
|
35
|
+
```json
|
|
36
|
+
{ "type": "evaluate", "value": "document.cookie = 'session_id=abc123; path=/; SameSite=Lax'" }
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## 5. API Headers
|
|
40
|
+
|
|
41
|
+
Override `fetch` to inject Authorization headers (useful for `--test-type api`):
|
|
42
|
+
```json
|
|
43
|
+
{ "type": "evaluate", "value": "const orig = window.fetch; window.fetch = (url, opts = {}) => { opts.headers = { ...opts.headers, 'Authorization': 'Bearer eyJhbG...' }; return orig(url, opts); }" }
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## 6. OAuth/SSO
|
|
47
|
+
|
|
48
|
+
Use a test-environment bypass endpoint, pre-authenticated token injection, or CI-obtained session cookie. External OAuth providers (Google, GitHub, Okta) can't be automated directly.
|
|
49
|
+
|
|
50
|
+
## Auth Auto-Login
|
|
51
|
+
|
|
52
|
+
Automatically fetch an auth token from a login API endpoint before tests run. Avoids repeating login flows in every test.
|
|
53
|
+
|
|
54
|
+
### Config fields
|
|
55
|
+
|
|
56
|
+
| Field | Description | Env / CLI |
|
|
57
|
+
|-------|-------------|-----------|
|
|
58
|
+
| `authLoginEndpoint` | Full URL to POST credentials to | `AUTH_LOGIN_ENDPOINT` / `--auth-login-endpoint` |
|
|
59
|
+
| `authCredentials` | Object with login credentials (config file only — never in env vars) | — |
|
|
60
|
+
| `authTokenPath` | Dot-path to extract token from response JSON (default: `'token'`) | `AUTH_TOKEN_PATH` / `--auth-token-path` |
|
|
61
|
+
|
|
62
|
+
### Flow
|
|
63
|
+
|
|
64
|
+
1. Before workers start, if `authLoginEndpoint` is set and `authToken` is NOT already set, `fetchAuthToken()` POSTs `authCredentials` as JSON.
|
|
65
|
+
2. The token is extracted from the response using `authTokenPath` (supports dot notation, e.g. `'data.access_token'`).
|
|
66
|
+
3. The extracted token is stored in `config.authToken`.
|
|
67
|
+
4. Each test worker navigates to `baseUrl` and injects `localStorage[authStorageKey] = authToken` before `beforeEach` hooks run.
|
|
68
|
+
|
|
69
|
+
### Example config
|
|
70
|
+
|
|
71
|
+
```js
|
|
72
|
+
export default {
|
|
73
|
+
authLoginEndpoint: 'https://api.example.com/auth/login',
|
|
74
|
+
authCredentials: { email: 'test@example.com', password: 'secret123' },
|
|
75
|
+
authTokenPath: 'data.access_token',
|
|
76
|
+
authStorageKey: 'accessToken',
|
|
77
|
+
};
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
If `authToken` is already set (via config, env var, or CLI), the auto-login step is skipped.
|
|
81
|
+
|
|
82
|
+
## Clearing State
|
|
83
|
+
|
|
84
|
+
Each test runs in a fresh browser context (auto-clean). Use `clear_cookies` to explicitly clear cookies + localStorage + sessionStorage mid-test.
|
|
85
|
+
|
|
86
|
+
## Reusable Auth Modules
|
|
87
|
+
|
|
88
|
+
Create `e2e/modules/login.json` or `e2e/modules/auth-token.json` and reference via:
|
|
89
|
+
```json
|
|
90
|
+
{ "$use": "login", "params": { "email": "...", "password": "..." } }
|
|
91
|
+
```
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# GraphQL Action Reference
|
|
2
|
+
|
|
3
|
+
The `gql` action executes GraphQL queries and mutations via browser `fetch`, with automatic auth token injection from `localStorage`. It also installs a `window.__e2eGql(query, vars)` helper for use in subsequent `evaluate` actions.
|
|
4
|
+
|
|
5
|
+
## Action Fields
|
|
6
|
+
|
|
7
|
+
| Action | Fields | Behavior |
|
|
8
|
+
|--------|--------|----------|
|
|
9
|
+
| `gql` | `value` (query string, required), `text` (variables JSON, optional), `selector` (assertion JS expression, optional) | Sends a GraphQL request to the configured endpoint. Reads auth token from localStorage. Throws on GraphQL errors. Returns `{ value: response.data }`. Stores full response on `window.__e2eLastGql`. |
|
|
10
|
+
|
|
11
|
+
## Config Fields
|
|
12
|
+
|
|
13
|
+
| Field | Default | Env Var | CLI Flag |
|
|
14
|
+
|-------|---------|---------|----------|
|
|
15
|
+
| `gqlEndpoint` | `/api/graphql` | `GQL_ENDPOINT` | `--gql-endpoint` |
|
|
16
|
+
| `gqlAuthHeader` | `Authorization` | `GQL_AUTH_HEADER` | `--gql-auth-header` |
|
|
17
|
+
| `gqlAuthKey` | `accessToken` | `GQL_AUTH_KEY` | `--gql-auth-key` |
|
|
18
|
+
| `gqlAuthPrefix` | `Bearer ` | `GQL_AUTH_PREFIX` | `--gql-auth-prefix` |
|
|
19
|
+
|
|
20
|
+
The endpoint path is appended to `location.origin`. The auth token is read from `localStorage[gqlAuthKey]` and sent as `gqlAuthHeader: gqlAuthPrefix + token`.
|
|
21
|
+
|
|
22
|
+
## Examples
|
|
23
|
+
|
|
24
|
+
### Basic query
|
|
25
|
+
```json
|
|
26
|
+
{ "type": "gql", "value": "{ users { id name } }" }
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
### With variables
|
|
30
|
+
```json
|
|
31
|
+
{ "type": "gql", "value": "query($id: ID, $orgId: ID) { orders(userId: $id, orgId: $orgId) { orderId status } }", "text": "{\"id\": \"abc-123\", \"orgId\": \"org-456\"}" }
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
### With inline assertion (selector field)
|
|
35
|
+
```json
|
|
36
|
+
// selector is a JS expression where `r` is the full GraphQL response
|
|
37
|
+
{ "type": "gql", "value": "{ pendingOrders(userId: \"abc-123\") { status } }", "selector": "r.data.pendingOrders.some(o => o.status === 'CANCELLED') ? 'FAIL: cancelled order found in pending list' : 'OK: all pending'" }
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### Using the installed helper in evaluate
|
|
41
|
+
```json
|
|
42
|
+
// After any gql action runs, window.__e2eGql(query, vars) is available
|
|
43
|
+
{ "type": "gql", "value": "{ __typename }" }
|
|
44
|
+
{ "type": "evaluate", "value": "(async () => { const r = await window.__e2eGql('query { orders(status: [PROCESSING]) { orderId } }'); for (const o of r.data.orders) await window.__e2eGql('mutation($id: ID, $input: OrderInput) { updateOrder(orderId: $id, input: $input) { orderId } }', { id: o.orderId, input: { status: 'COMPLETED' } }); return 'Updated ' + r.data.orders.length; })()" }
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Custom Auth Header Config
|
|
48
|
+
|
|
49
|
+
If your API uses a non-standard auth header (e.g., `x-api-key` instead of `Authorization`):
|
|
50
|
+
|
|
51
|
+
```js
|
|
52
|
+
// e2e.config.js
|
|
53
|
+
export default {
|
|
54
|
+
gqlEndpoint: '/api/graphql',
|
|
55
|
+
gqlAuthHeader: 'x-api-key', // custom header name
|
|
56
|
+
gqlAuthKey: 'apiToken', // localStorage key to read from
|
|
57
|
+
gqlAuthPrefix: '', // no 'Bearer ' prefix — raw token
|
|
58
|
+
};
|
|
59
|
+
```
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Issue-to-Test Verification Reference
|
|
2
|
+
|
|
3
|
+
Turns bug reports and feature requests into executable E2E tests.
|
|
4
|
+
|
|
5
|
+
## Supported Providers
|
|
6
|
+
|
|
7
|
+
- **GitHub** (`github.com`) — requires `gh` CLI (`gh auth login`)
|
|
8
|
+
- **GitLab** (including self-hosted) — requires `glab` CLI (`glab auth login`)
|
|
9
|
+
|
|
10
|
+
Auto-detected from URL. All external commands use `execFileSync` (no shell injection).
|
|
11
|
+
|
|
12
|
+
## Two AI Modes
|
|
13
|
+
|
|
14
|
+
### 1. Prompt Mode (default, no API key)
|
|
15
|
+
|
|
16
|
+
`e2e_issue` MCP tool returns issue details + a structured prompt. Claude Code then uses `e2e_create_test` to create tests and `e2e_run` to execute them.
|
|
17
|
+
|
|
18
|
+
### 2. Verify Mode (requires `ANTHROPIC_API_KEY`)
|
|
19
|
+
|
|
20
|
+
Calls Claude API directly to generate tests, runs them, and reports whether the bug is confirmed or not reproducible.
|
|
21
|
+
|
|
22
|
+
## Config Fields
|
|
23
|
+
|
|
24
|
+
| Field | Description | Env Var |
|
|
25
|
+
|-------|-------------|---------|
|
|
26
|
+
| `anthropicApiKey` | Required for verify/generate mode | `ANTHROPIC_API_KEY` |
|
|
27
|
+
| `anthropicModel` | Claude model for generation (default: `claude-sonnet-4-5-20250929`) | `ANTHROPIC_MODEL` |
|
|
28
|
+
|
|
29
|
+
## Bug Verification Logic
|
|
30
|
+
|
|
31
|
+
Generated tests assert **correct** behavior:
|
|
32
|
+
- **Test failure** = bug confirmed (expected behavior doesn't match reality)
|
|
33
|
+
- **All tests pass** = not reproducible
|
|
34
|
+
|
|
35
|
+
## Test Categories (`testType`)
|
|
36
|
+
|
|
37
|
+
The `--test-type` CLI flag (or `testType` MCP parameter) controls what kind of tests the AI generates:
|
|
38
|
+
|
|
39
|
+
- **`e2e`** (default): UI-driven tests — navigate pages, interact with elements (click, type, select), verify visible state. Never uses `evaluate` for API calls.
|
|
40
|
+
- **`api`**: Backend API tests — use `evaluate` actions for GraphQL/REST calls, assert response shapes and values. No UI interaction needed.
|
|
41
|
+
|
|
42
|
+
CLI: `e2e-runner issue <url> --generate --test-type api`
|
|
43
|
+
MCP: `e2e_issue({ url, testType: "api" })`
|
|
44
|
+
|
|
45
|
+
## GitLab Limitations
|
|
46
|
+
|
|
47
|
+
- Requires `glab` CLI installed and authenticated (`glab auth login`)
|
|
48
|
+
- Self-hosted GitLab instances supported via URL detection
|
|
49
|
+
- Verify mode works with GitLab issues but does NOT post comments back to the issue
|
|
50
|
+
- Private repos require `glab` to be authenticated with appropriate access
|
|
51
|
+
- The `authToken`/`authStorageKey` params on `e2e_issue` are for the **app under test**, not for GitLab API auth
|
|
52
|
+
|
|
53
|
+
## Key Files
|
|
54
|
+
|
|
55
|
+
| File | Purpose |
|
|
56
|
+
|------|---------|
|
|
57
|
+
| `src/issues.js` | GitHub/GitLab provider drivers |
|
|
58
|
+
| `src/ai-generate.js` | AI prompt builder + Claude API |
|
|
59
|
+
| `src/verify.js` | Verification orchestrator: fetch + generate + run |
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Multi-Pool Support
|
|
2
|
+
|
|
3
|
+
A single runner can distribute tests across multiple Chrome pools on different machines. `src/pool-manager.js` abstracts pool selection behind a **least-pressure** strategy. `poolUrl` (singular string) remains fully backwards compatible.
|
|
4
|
+
|
|
5
|
+
## Configuration
|
|
6
|
+
|
|
7
|
+
```js
|
|
8
|
+
// Single pool (unchanged)
|
|
9
|
+
export default { poolUrl: 'ws://localhost:3333' };
|
|
10
|
+
|
|
11
|
+
// Multi-pool — tests distribute across all pools
|
|
12
|
+
export default {
|
|
13
|
+
poolUrls: ['ws://machine1:3333', 'ws://machine2:3333'],
|
|
14
|
+
concurrency: 15,
|
|
15
|
+
};
|
|
16
|
+
|
|
17
|
+
// CI environment profile
|
|
18
|
+
export default {
|
|
19
|
+
poolUrl: 'ws://localhost:3333',
|
|
20
|
+
environments: {
|
|
21
|
+
ci: { poolUrls: ['ws://ci-chrome1:3333', 'ws://ci-chrome2:3333'], concurrency: 10 },
|
|
22
|
+
},
|
|
23
|
+
};
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
**CLI:** `e2e-runner run --all --pool-urls ws://m1:3333,ws://m2:3333`
|
|
27
|
+
**Env:** `CHROME_POOL_URLS=ws://m1:3333,ws://m2:3333` (comma-separated)
|
|
28
|
+
|
|
29
|
+
## Pool Selection Algorithm (`selectPool`)
|
|
30
|
+
|
|
31
|
+
1. Query all pools' `/pressure` in parallel
|
|
32
|
+
2. Filter to reachable pools with `running < maxConcurrent`
|
|
33
|
+
3. Sort by: lowest `running/maxConcurrent` → fewest queued → most free slots
|
|
34
|
+
4. Return best candidate URL
|
|
35
|
+
5. If all full, poll every 2s up to 60s, then pick least-pressured anyway
|
|
36
|
+
|
|
37
|
+
## Pool-Aware Queue
|
|
38
|
+
|
|
39
|
+
Before opening a browser connection, each worker checks the pool's `/pressure` endpoint. If the pool is at capacity, the worker waits (polling every 2s, up to 60s) for a free slot instead of piling requests into browserless's internal queue. This prevents memory pressure and SIGKILL of Chrome processes under heavy load.
|
|
40
|
+
|
|
41
|
+
## Failure Resilience
|
|
42
|
+
|
|
43
|
+
Dead pools are excluded from selection. `waitForAnyPool` succeeds if any pool responds. Failed connections trigger re-selection on the next worker.
|
|
44
|
+
|
|
45
|
+
## Config Normalization in `loadConfig()`
|
|
46
|
+
|
|
47
|
+
- If `poolUrls` array is set → `config._poolUrls = poolUrls`, `config.poolUrl = poolUrls[0]`
|
|
48
|
+
- Else → `config._poolUrls = [config.poolUrl]`
|
|
49
|
+
- `config.poolUrls` is deleted after normalization (use `config._poolUrls` internally)
|
|
50
|
+
|
|
51
|
+
## Key Functions (`src/pool-manager.js`)
|
|
52
|
+
|
|
53
|
+
| Function | Purpose |
|
|
54
|
+
|----------|---------|
|
|
55
|
+
| `getPoolUrls(config)` | Returns `config._poolUrls` (always an array) |
|
|
56
|
+
| `getAllPoolStatuses(poolUrls)` | Fetches `/pressure` from all pools in parallel |
|
|
57
|
+
| `getAggregatedPoolStatus(poolUrls)` | Combined view: `totalRunning`, `totalMaxConcurrent`, per-pool details |
|
|
58
|
+
| `waitForAnyPool(poolUrls)` | Blocks until at least one pool is reachable + available |
|
|
59
|
+
| `selectPool(poolUrls)` | Picks pool with lowest pressure ratio |
|
|
60
|
+
| `selectAndConnect(config)` | Convenience: selectPool + connectToPool in one call |
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Network Debugging Reference
|
|
2
|
+
|
|
3
|
+
## Network Error Handling
|
|
4
|
+
|
|
5
|
+
### `assert_no_network_errors` action
|
|
6
|
+
|
|
7
|
+
Checks accumulated `requestfailed` events during the test. If any network errors exist (e.g., `net::ERR_CONNECTION_REFUSED`), the test fails with details of each error URL. Place after critical page loads.
|
|
8
|
+
|
|
9
|
+
### `failOnNetworkError` config option
|
|
10
|
+
|
|
11
|
+
When `true`, automatically fails any test that has network errors after all actions complete. Default: `false` (opt-in).
|
|
12
|
+
|
|
13
|
+
Set via: config `failOnNetworkError: true` | CLI `--fail-on-network-error` | env `FAIL_ON_NETWORK_ERROR=true` | MCP `failOnNetworkError: true`
|
|
14
|
+
|
|
15
|
+
### `networkIgnoreDomains` config option
|
|
16
|
+
|
|
17
|
+
Array of domain substrings to filter from network error tracking. Errors from matching URLs are silently dropped by both `assert_no_network_errors` and `failOnNetworkError`.
|
|
18
|
+
|
|
19
|
+
Set via: config `networkIgnoreDomains: ['google-analytics.com', 'fonts.googleapis.com']` | CLI `--network-ignore-domains ga.com,fonts.com` | env `NETWORK_IGNORE_DOMAINS=ga.com,fonts.com` (comma-separated)
|
|
20
|
+
|
|
21
|
+
## Network Request/Response Logging
|
|
22
|
+
|
|
23
|
+
All XHR/fetch requests are captured with full detail regardless of status code:
|
|
24
|
+
|
|
25
|
+
- `url`, `method`, `status`, `statusText`, `duration`
|
|
26
|
+
- `requestHeaders` — all request headers as object
|
|
27
|
+
- `requestBody` — POST body (from `req.postData()`)
|
|
28
|
+
- `responseHeaders` — all response headers as object
|
|
29
|
+
- `responseBody` — full response text (truncated at 50KB)
|
|
30
|
+
|
|
31
|
+
Response bodies are read asynchronously and flushed via `Promise.allSettled` before the browser disconnects. Data is stored in the `network_logs` column in SQLite and displayed in the dashboard.
|
|
32
|
+
|
|
33
|
+
## MCP Response Optimization
|
|
34
|
+
|
|
35
|
+
The `e2e_run` MCP tool returns a compact `networkSummary` instead of full logs (~5KB vs ~400KB):
|
|
36
|
+
|
|
37
|
+
```json
|
|
38
|
+
{
|
|
39
|
+
"networkSummary": [{
|
|
40
|
+
"name": "test-name",
|
|
41
|
+
"totalRequests": 37,
|
|
42
|
+
"statusDistribution": { "2xx": 30, "3xx": 5, "4xx": 1, "5xx": 0, "other": 1 },
|
|
43
|
+
"avgDurationMs": 245,
|
|
44
|
+
"failedRequests": [{ "url": "/api/x", "method": "POST", "status": 500 }],
|
|
45
|
+
"slowestRequests": [{ "url": "/api/y", "method": "GET", "status": 200, "duration": 1200 }]
|
|
46
|
+
}]
|
|
47
|
+
}
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Drill-Down Pattern
|
|
51
|
+
|
|
52
|
+
The response includes `runDbId` — the SQLite row ID. Use it with `e2e_network_logs` to drill down:
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
1. e2e_run → compact summary + runDbId
|
|
56
|
+
2. e2e_network_logs(runDbId) → all requests (url, method, status, duration)
|
|
57
|
+
3. e2e_network_logs(runDbId, errorsOnly: true) → only failed requests
|
|
58
|
+
4. e2e_network_logs(runDbId, includeHeaders: true) → with headers
|
|
59
|
+
5. e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Dashboard REST equivalent: `GET /api/db/runs/:id/network-logs?testName=X&errorsOnly=true&includeHeaders=true`
|
|
@@ -150,6 +150,10 @@ export default {
|
|
|
150
150
|
|
|
151
151
|
Activate with `--env staging` or `E2E_ENV=staging`. Profile values override all other config.
|
|
152
152
|
|
|
153
|
+
## CI Output Formats
|
|
154
|
+
|
|
155
|
+
Use `--output <format>` to control report output: `json` (default), `junit` (JUnit XML), or `both`. JUnit XML is generated without external dependencies and saved to `{screenshotsDir}/junit.xml`. The `generateJUnitXML()` function is also exported from the programmatic API.
|
|
156
|
+
|
|
153
157
|
## Config Priority (ascending)
|
|
154
158
|
|
|
155
159
|
1. Hardcoded defaults
|