@matware/e2e-runner 1.2.1 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/.claude-plugin/marketplace.json +21 -0
  2. package/.mcp.json +2 -2
  3. package/.opencode/commands/create-test.md +63 -0
  4. package/.opencode/commands/run.md +50 -0
  5. package/.opencode/commands/verify-issue.md +62 -0
  6. package/.opencode/skills/e2e-testing/SKILL.md +181 -0
  7. package/.opencode/skills/e2e-testing/references/action-types.md +143 -0
  8. package/.opencode/skills/e2e-testing/references/auth-strategies.md +91 -0
  9. package/.opencode/skills/e2e-testing/references/graphql.md +59 -0
  10. package/.opencode/skills/e2e-testing/references/issue-verification.md +59 -0
  11. package/.opencode/skills/e2e-testing/references/multi-pool.md +60 -0
  12. package/.opencode/skills/e2e-testing/references/network-debugging.md +62 -0
  13. package/.opencode/skills/e2e-testing/references/test-json-format.md +163 -0
  14. package/.opencode/skills/e2e-testing/references/troubleshooting.md +224 -0
  15. package/.opencode/skills/e2e-testing/references/variables.md +41 -0
  16. package/.opencode/skills/e2e-testing/references/visual-verification.md +89 -0
  17. package/OPENCODE.md +166 -0
  18. package/README.md +581 -55
  19. package/agents/test-creator.md +54 -1
  20. package/agents/test-improver.md +37 -0
  21. package/bin/cli.js +408 -16
  22. package/commands/create-test.md +16 -1
  23. package/opencode.json +11 -0
  24. package/package.json +7 -2
  25. package/scripts/setup-opencode.sh +113 -0
  26. package/skills/e2e-testing/SKILL.md +10 -3
  27. package/skills/e2e-testing/references/action-types.md +48 -5
  28. package/skills/e2e-testing/references/auth-strategies.md +91 -0
  29. package/skills/e2e-testing/references/graphql.md +59 -0
  30. package/skills/e2e-testing/references/issue-verification.md +59 -0
  31. package/skills/e2e-testing/references/multi-pool.md +60 -0
  32. package/skills/e2e-testing/references/network-debugging.md +62 -0
  33. package/skills/e2e-testing/references/test-json-format.md +4 -0
  34. package/skills/e2e-testing/references/troubleshooting.md +44 -2
  35. package/skills/e2e-testing/references/variables.md +41 -0
  36. package/skills/e2e-testing/references/visual-verification.md +89 -0
  37. package/src/actions.js +324 -2
  38. package/src/ai-generate.js +58 -8
  39. package/src/config.js +143 -0
  40. package/src/dashboard.js +145 -13
  41. package/src/db.js +130 -2
  42. package/src/index.js +7 -6
  43. package/src/learner-sqlite.js +304 -0
  44. package/src/learner.js +8 -3
  45. package/src/mcp-tools.js +1121 -43
  46. package/src/module-resolver.js +37 -0
  47. package/src/narrate.js +37 -0
  48. package/src/pool-manager.js +223 -0
  49. package/src/reporter.js +82 -1
  50. package/src/runner.js +157 -28
  51. package/src/sync/auth.js +354 -0
  52. package/src/sync/client.js +572 -0
  53. package/src/sync/hub-routes.js +816 -0
  54. package/src/sync/index.js +68 -0
  55. package/src/sync/middleware.js +347 -0
  56. package/src/sync/queue.js +209 -0
  57. package/src/sync/schema.js +540 -0
  58. package/src/verify.js +10 -7
  59. package/src/watch.js +384 -0
  60. package/templates/build-dashboard.js +47 -6
  61. package/templates/dashboard/js/api.js +60 -0
  62. package/templates/dashboard/js/init.js +13 -0
  63. package/templates/dashboard/js/keyboard.js +46 -0
  64. package/templates/dashboard/js/state.js +40 -0
  65. package/templates/dashboard/js/toast.js +41 -0
  66. package/templates/dashboard/js/utils.js +196 -0
  67. package/templates/dashboard/js/view-live.js +143 -0
  68. package/templates/dashboard/js/view-runs.js +572 -0
  69. package/templates/dashboard/js/view-tests.js +294 -0
  70. package/templates/dashboard/js/view-watch.js +242 -0
  71. package/templates/dashboard/js/websocket.js +110 -0
  72. package/templates/dashboard/styles/base.css +69 -0
  73. package/templates/dashboard/styles/components.css +110 -0
  74. package/templates/dashboard/styles/view-live.css +74 -0
  75. package/templates/dashboard/styles/view-runs.css +207 -0
  76. package/templates/dashboard/styles/view-tests.css +96 -0
  77. package/templates/dashboard/styles/view-watch.css +53 -0
  78. package/templates/dashboard/template.html +165 -99
  79. package/templates/dashboard.html +1596 -541
  80. package/templates/sample-test.json +0 -8
  81. package/templates/dashboard/app.js +0 -1152
  82. package/templates/dashboard/styles.css +0 -413
@@ -37,11 +37,26 @@ Help the user create a new E2E test file by exploring the application and design
37
37
  - Add `wait` actions before assertions on dynamic content
38
38
  - Add `assert_no_network_errors` after critical page loads
39
39
  - Consider adding an `expect` field for visual verification
40
+ - **DRY**: If auth/setup is repeated across tests, use `beforeEach` hook (object format) instead of repeating per test
41
+ - **DRY**: If 3+ tests repeat the same action pattern, create a module with `e2e_create_module` first
40
42
 
41
- 7. **Create the test** — Call `e2e_create_test` with the designed test structure. Consider creating reusable modules with `e2e_create_module` for repeated sequences (auth, navigation).
43
+ 7. **Create the test** — Call `e2e_create_test` with the designed test structure.
44
+ - Check existing modules (`e2e/modules/`) — reuse them via `$use` instead of duplicating actions
45
+ - Create new modules with `e2e_create_module` for repeated sequences (auth, navigation, screenshot patterns)
46
+ - Use object format `{ "beforeEach": [...], "tests": [...] }` when hooks are needed
42
47
 
43
48
  8. **Validate** — Run the newly created test with `e2e_run` using the `suite` parameter. Analyze results and iterate if needed.
44
49
 
50
+ ## Naming Rules (CRITICAL)
51
+
52
+ Suite names MUST be unique and specific to the feature, issue, or user flow:
53
+ - GOOD: `login-valid-credentials`, `issue-1743-auth-redirect`, `checkout-payment-flow`
54
+ - BAD: `all`, `test`, `debug`, `new`, `temp`, `main`, `suite`
55
+
56
+ If testing a GitHub/GitLab issue, include the issue number: `issue-1743-auth-timeout`, `bug-502-duplicate-submit`
57
+
58
+ Before creating, always call `e2e_list` to verify the name doesn't already exist.
59
+
45
60
  ## Arguments
46
61
 
47
62
  The user may provide:
package/opencode.json ADDED
@@ -0,0 +1,11 @@
1
+ {
2
+ "$schema": "https://opencode.ai/schemas/opencode.json",
3
+ "mcp": {
4
+ "e2e-runner": {
5
+ "type": "local",
6
+ "command": "node",
7
+ "args": ["bin/mcp-server.js"],
8
+ "cwd": "${workspaceFolder}"
9
+ }
10
+ }
11
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@matware/e2e-runner",
3
- "version": "1.2.1",
3
+ "version": "1.3.0",
4
4
  "mcpName": "io.github.fastslack/e2e-runner",
5
5
  "description": "E2E test runner using Chrome Pool (browserless/chrome) with parallel execution",
6
6
  "type": "module",
@@ -20,7 +20,11 @@
20
20
  ".mcp.json",
21
21
  "skills/",
22
22
  "commands/",
23
- "agents/"
23
+ "agents/",
24
+ ".opencode/",
25
+ "opencode.json",
26
+ "OPENCODE.md",
27
+ "scripts/setup-opencode.sh"
24
28
  ],
25
29
  "keywords": [
26
30
  "e2e",
@@ -31,6 +35,7 @@
31
35
  "parallel",
32
36
  "mcp",
33
37
  "claude-code",
38
+ "opencode",
34
39
  "github-issues",
35
40
  "ai-testing"
36
41
  ],
@@ -0,0 +1,113 @@
1
+ #!/usr/bin/env bash
2
+ # Setup @matware/e2e-runner for OpenCode
3
+ # Usage: ./setup-opencode.sh [--global]
4
+
5
+ set -e
6
+
7
+ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
8
+ PACKAGE_DIR="$(dirname "$SCRIPT_DIR")"
9
+
10
+ # Colors
11
+ RED='\033[0;31m'
12
+ GREEN='\033[0;32m'
13
+ YELLOW='\033[1;33m'
14
+ NC='\033[0m' # No Color
15
+
16
+ log_info() { echo -e "${GREEN}✓${NC} $1"; }
17
+ log_warn() { echo -e "${YELLOW}!${NC} $1"; }
18
+ log_error() { echo -e "${RED}✗${NC} $1"; }
19
+
20
+ # Check if --global flag is passed
21
+ GLOBAL_INSTALL=false
22
+ if [[ "$1" == "--global" ]]; then
23
+ GLOBAL_INSTALL=true
24
+ fi
25
+
26
+ if [ "$GLOBAL_INSTALL" = true ]; then
27
+ echo "Setting up e2e-runner for OpenCode (global)..."
28
+
29
+ # Global config directory
30
+ OPENCODE_CONFIG="${HOME}/.config/opencode"
31
+
32
+ # Create directories
33
+ mkdir -p "${OPENCODE_CONFIG}/skills"
34
+ mkdir -p "${OPENCODE_CONFIG}/commands"
35
+
36
+ # Copy skills
37
+ if [ -d "${PACKAGE_DIR}/.opencode/skills" ]; then
38
+ cp -r "${PACKAGE_DIR}/.opencode/skills/"* "${OPENCODE_CONFIG}/skills/"
39
+ log_info "Installed skills to ${OPENCODE_CONFIG}/skills/"
40
+ fi
41
+
42
+ # Copy commands
43
+ if [ -d "${PACKAGE_DIR}/.opencode/commands" ]; then
44
+ cp -r "${PACKAGE_DIR}/.opencode/commands/"* "${OPENCODE_CONFIG}/commands/"
45
+ log_info "Installed commands to ${OPENCODE_CONFIG}/commands/"
46
+ fi
47
+
48
+ echo ""
49
+ log_warn "To use the MCP server, add this to your project's opencode.json:"
50
+ echo ""
51
+ cat << 'EOF'
52
+ {
53
+ "mcp": {
54
+ "e2e-runner": {
55
+ "type": "local",
56
+ "command": "e2e-runner-mcp"
57
+ }
58
+ }
59
+ }
60
+ EOF
61
+ echo ""
62
+
63
+ else
64
+ echo "Setting up e2e-runner for OpenCode (project-local)..."
65
+
66
+ # Check if we're in the package directory or a project that installed it
67
+ if [ -f "${PWD}/opencode.json" ] && [ -d "${PWD}/.opencode" ]; then
68
+ # We're in the package directory
69
+ log_info "Already in e2e-runner package directory"
70
+ exit 0
71
+ fi
72
+
73
+ # Look for node_modules/@matware/e2e-runner
74
+ if [ -d "${PWD}/node_modules/@matware/e2e-runner" ]; then
75
+ PACKAGE_DIR="${PWD}/node_modules/@matware/e2e-runner"
76
+ elif [ ! -f "${PACKAGE_DIR}/package.json" ]; then
77
+ log_error "Cannot find @matware/e2e-runner package"
78
+ log_error "Run: npm install @matware/e2e-runner"
79
+ exit 1
80
+ fi
81
+
82
+ # Copy opencode.json if it doesn't exist
83
+ if [ ! -f "${PWD}/opencode.json" ]; then
84
+ cp "${PACKAGE_DIR}/opencode.json" "${PWD}/opencode.json"
85
+ log_info "Created opencode.json"
86
+ else
87
+ log_warn "opencode.json already exists - merge manually if needed"
88
+ fi
89
+
90
+ # Copy .opencode directory
91
+ mkdir -p "${PWD}/.opencode"
92
+
93
+ if [ -d "${PACKAGE_DIR}/.opencode/skills" ]; then
94
+ mkdir -p "${PWD}/.opencode/skills"
95
+ cp -r "${PACKAGE_DIR}/.opencode/skills/"* "${PWD}/.opencode/skills/"
96
+ log_info "Installed skills to .opencode/skills/"
97
+ fi
98
+
99
+ if [ -d "${PACKAGE_DIR}/.opencode/commands" ]; then
100
+ mkdir -p "${PWD}/.opencode/commands"
101
+ cp -r "${PACKAGE_DIR}/.opencode/commands/"* "${PWD}/.opencode/commands/"
102
+ log_info "Installed commands to .opencode/commands/"
103
+ fi
104
+ fi
105
+
106
+ echo ""
107
+ log_info "OpenCode setup complete!"
108
+ echo ""
109
+ echo "Next steps:"
110
+ echo " 1. Start the Chrome pool: npx e2e-runner pool start"
111
+ echo " 2. Restart OpenCode to load the MCP server"
112
+ echo " 3. Ask: 'Run all E2E tests'"
113
+ echo ""
@@ -85,7 +85,7 @@ Add an `expect` field to any test for AI-powered visual verification:
85
85
  ```json
86
86
  {
87
87
  "name": "dashboard-loads",
88
- "expect": "Should show patient list with at least 3 rows and no error messages",
88
+ "expect": "Should show data table with at least 3 rows and no error messages",
89
89
  "actions": [...]
90
90
  }
91
91
  ```
@@ -162,5 +162,12 @@ Start/stop the web dashboard with `e2e_dashboard_start` / `e2e_dashboard_stop` f
162
162
  ## References
163
163
 
164
164
  - [Action Types Reference](references/action-types.md) — Complete catalog of 28+ action types with fields and examples
165
- - [Test JSON Format](references/test-json-format.md) — JSON structure, hooks, serial, retries, modules, exclude patterns, environment profiles
166
- - [Troubleshooting](references/troubleshooting.md) — Common problems and solutions
165
+ - [Test JSON Format](references/test-json-format.md) — JSON structure, hooks, serial, retries, modules, exclude patterns, environment profiles, CI output
166
+ - [GraphQL Action](references/graphql.md) — GQL action config, variables, inline assertions, __e2eGql helper
167
+ - [Authentication Strategies](references/auth-strategies.md) — 6 auth methods + auto-login + reusable auth modules
168
+ - [Network Debugging](references/network-debugging.md) — Error handling, request logging, drill-down pattern
169
+ - [Visual Verification](references/visual-verification.md) — Expect field, double screenshots, strictness levels, verdict format
170
+ - [Multi-Pool Support](references/multi-pool.md) — Config, selection algorithm, failover, pool-aware queue
171
+ - [Variables](references/variables.md) — SQLite-backed variables, syntax, MCP tool, dashboard UI, REST API
172
+ - [Issue Verification](references/issue-verification.md) — GitHub/GitLab, AI modes, test categories, GitLab limitations
173
+ - [Troubleshooting](references/troubleshooting.md) — Common problems, pre-validation, screenshot hashes, dashboard
@@ -31,6 +31,21 @@ Complete catalog of all action types supported by @matware/e2e-runner.
31
31
  | `focus_autocomplete` | `text` (label text) | Focus an autocomplete input by label. Supports MUI `.MuiAutocomplete-root` and `[role="combobox"]`. |
32
32
  | `click_chip` | `text` | Click a chip/tag element by text. Searches `[class*="Chip"]`, `[class*="chip"]`, `[data-chip]`. |
33
33
 
34
+ ## Storage
35
+
36
+ | Action | Fields | Description |
37
+ |--------|--------|-------------|
38
+ | `set_storage` | `value` (`"key=val"`), `selector` (`"session"` optional) | Set a `localStorage` key (default) or `sessionStorage` key (with `selector: "session"`). |
39
+ | `assert_storage` | `value` (`"key"` or `"key=expected"`), `selector` (`"session"` optional) | Without `=`: checks key exists. With `=`: checks exact value match. Uses `localStorage` by default, `sessionStorage` with `selector: "session"`. |
40
+
41
+ ## Smart Interaction
42
+
43
+ | Action | Fields | Description |
44
+ |--------|--------|-------------|
45
+ | `click_icon` | `value` (icon identifier), `selector` (scope, optional) | Click an icon by `data-testid`, `data-icon`, `aria-label`, CSS class, or SVG title. Walks up to nearest clickable ancestor (`button`, `a`, `[role="button"]`). Works with MUI, FontAwesome, Heroicons, Bootstrap Icons, Lucide. |
46
+ | `click_menu_item` | `text` (menu item text), `selector` (scope, optional) | Click a menu item by text. Searches `[role="menuitem"]`, `[role="menuitemradio"]`, `[role="menuitemcheckbox"]`, `.dropdown-item`, `.menu-item`, `[class*="MenuItem"]`, `[role="menu"] > li`. Waits for element to appear. |
47
+ | `click_in_context` | `text` (container text), `selector` (child to click) | Find the smallest container whose text includes `text`, then click the `selector` child within it. Containers: `section`, `article`, `[class*="card"]`, `li`, `tr`, `div[class]`, etc. Both fields required. |
48
+
34
49
  ## Assertions
35
50
 
36
51
  | Action | Fields | Description |
@@ -61,9 +76,21 @@ Complete catalog of all action types supported by @matware/e2e-runner.
61
76
  | `get_text` | `selector` | Returns `{ value: textContent.trim() }`. Non-assertion — never fails. |
62
77
  | `screenshot` | `value` (filename, optional) | Captures screenshot. Filename gets timestamp suffix for uniqueness. |
63
78
  | `wait` | `selector` OR `text` OR `value` (ms) | Wait for selector, text on page, or fixed delay. |
64
- | `evaluate` | `value` (JS code) | Run JavaScript in browser context. **Strict**: returns starting with `FAIL:`/`ERROR:` test fails. Returns `false` test fails. |
79
+ | `wait_network_idle` | `value` (idle ms, default 500), `timeout` (max wait ms, default 30000) | Waits for all network requests to complete. Uses Puppeteer's `page.waitForNetworkIdle()`. Useful after SPA page transitions or data loading. |
80
+ | `evaluate` | `value` (JS code) | Run JavaScript in browser context. See **Strict Evaluate** below. |
65
81
  | `clear_cookies` | `value` (origin, optional) | Clears cookies, localStorage, sessionStorage for origin. |
66
82
 
83
+ ### Strict Evaluate Semantics
84
+
85
+ The `evaluate` action checks the return value:
86
+
87
+ - If the JS returns a string starting with `FAIL:`, `ERROR:`, or `FAILED:` → the test **fails** with that message.
88
+ - If the JS returns `false` → the test **fails** (`evaluate returned false`).
89
+ - If the JS returns any other non-null value → stored as `{ value: result }` for visibility.
90
+ - If the JS throws → the test **fails** (standard Puppeteer error).
91
+
92
+ This prevents false PASSes where evaluate actions return error strings that were previously silently ignored.
93
+
67
94
  ## Action-Level Retry
68
95
 
69
96
  Any action can have `"retries": N` for per-action retry on failure:
@@ -79,14 +106,14 @@ Delay between retries: `actionRetryDelay` config (default 500ms).
79
106
 
80
107
  ### React input + autocomplete flow
81
108
  ```json
82
- { "type": "focus_autocomplete", "text": "Diagnosis" },
83
- { "type": "type_react", "selector": "#diagnosis-input", "value": "Cefalea" },
84
- { "type": "click_option", "text": "Cefalea tensional" }
109
+ { "type": "focus_autocomplete", "text": "Category" },
110
+ { "type": "type_react", "selector": "#category-input", "value": "Electr" },
111
+ { "type": "click_option", "text": "Electronics" }
85
112
  ```
86
113
 
87
114
  ### Regex click (last match)
88
115
  ```json
89
- { "type": "click_regex", "text": "start encounter", "selector": "button", "value": "last" }
116
+ { "type": "click_regex", "text": "add to cart", "selector": "button", "value": "last" }
90
117
  ```
91
118
 
92
119
  ### Form validation assertions
@@ -98,3 +125,19 @@ Delay between retries: `actionRetryDelay` config (default 500ms).
98
125
  { "type": "assert_matches", "selector": ".phone", "value": "\\d{3}-\\d{3}-\\d{4}" },
99
126
  { "type": "assert_count", "selector": ".table-row", "value": ">3" }
100
127
  ```
128
+
129
+ ### Storage operations
130
+ ```json
131
+ { "type": "set_storage", "value": "authToken=eyJhbGciOiJIUzI1NiJ9..." },
132
+ { "type": "assert_storage", "value": "authToken" },
133
+ { "type": "set_storage", "value": "theme=dark", "selector": "session" },
134
+ { "type": "assert_storage", "value": "theme=dark", "selector": "session" }
135
+ ```
136
+
137
+ ### Icon, menu, and contextual clicks
138
+ ```json
139
+ { "type": "click_icon", "value": "edit" },
140
+ { "type": "click_icon", "value": "delete", "selector": ".user-card" },
141
+ { "type": "click_menu_item", "text": "Export as PDF" },
142
+ { "type": "click_in_context", "text": "John Doe", "selector": "button.edit" }
143
+ ```
@@ -0,0 +1,91 @@
1
+ # Authentication Strategies
2
+
3
+ Tests can authenticate using multiple strategies depending on the app's auth mechanism.
4
+
5
+ ## 1. UI Login (universal)
6
+
7
+ Fill the login form in `beforeEach`. Works with any auth system — the browser stores cookies/tokens automatically.
8
+
9
+ ```json
10
+ { "hooks": { "beforeEach": [
11
+ { "type": "goto", "value": "/login" },
12
+ { "type": "type", "selector": "#email", "value": "test@example.com" },
13
+ { "type": "type", "selector": "#password", "value": "secret" },
14
+ { "type": "click", "text": "Sign In" },
15
+ { "type": "wait", "selector": ".dashboard" }
16
+ ]}}
17
+ ```
18
+
19
+ ## 2. JWT Injection (SPAs)
20
+
21
+ Skip the login form by injecting a token into `localStorage` or `sessionStorage`:
22
+
23
+ ```json
24
+ { "type": "set_storage", "value": "accessToken=eyJhbG..." }
25
+ { "type": "set_storage", "value": "token=eyJhbG...", "selector": "session" }
26
+ ```
27
+
28
+ ## 3. Config-Level Token
29
+
30
+ Set `authToken` + `authStorageKey` in config, env vars (`AUTH_TOKEN`, `AUTH_STORAGE_KEY`), or CLI flags (`--auth-token`, `--auth-storage-key`). Used by `e2e_capture` and `e2e_issue --verify`.
31
+
32
+ ## 4. Cookie-Based (server-rendered)
33
+
34
+ Set cookies via `evaluate`:
35
+ ```json
36
+ { "type": "evaluate", "value": "document.cookie = 'session_id=abc123; path=/; SameSite=Lax'" }
37
+ ```
38
+
39
+ ## 5. API Headers
40
+
41
+ Override `fetch` to inject Authorization headers (useful for `--test-type api`):
42
+ ```json
43
+ { "type": "evaluate", "value": "const orig = window.fetch; window.fetch = (url, opts = {}) => { opts.headers = { ...opts.headers, 'Authorization': 'Bearer eyJhbG...' }; return orig(url, opts); }" }
44
+ ```
45
+
46
+ ## 6. OAuth/SSO
47
+
48
+ Use a test-environment bypass endpoint, pre-authenticated token injection, or CI-obtained session cookie. External OAuth providers (Google, GitHub, Okta) can't be automated directly.
49
+
50
+ ## Auth Auto-Login
51
+
52
+ Automatically fetch an auth token from a login API endpoint before tests run. Avoids repeating login flows in every test.
53
+
54
+ ### Config fields
55
+
56
+ | Field | Description | Env / CLI |
57
+ |-------|-------------|-----------|
58
+ | `authLoginEndpoint` | Full URL to POST credentials to | `AUTH_LOGIN_ENDPOINT` / `--auth-login-endpoint` |
59
+ | `authCredentials` | Object with login credentials (config file only — never in env vars) | — |
60
+ | `authTokenPath` | Dot-path to extract token from response JSON (default: `'token'`) | `AUTH_TOKEN_PATH` / `--auth-token-path` |
61
+
62
+ ### Flow
63
+
64
+ 1. Before workers start, if `authLoginEndpoint` is set and `authToken` is NOT already set, `fetchAuthToken()` POSTs `authCredentials` as JSON.
65
+ 2. The token is extracted from the response using `authTokenPath` (supports dot notation, e.g. `'data.access_token'`).
66
+ 3. The extracted token is stored in `config.authToken`.
67
+ 4. Each test worker navigates to `baseUrl` and injects `localStorage[authStorageKey] = authToken` before `beforeEach` hooks run.
68
+
69
+ ### Example config
70
+
71
+ ```js
72
+ export default {
73
+ authLoginEndpoint: 'https://api.example.com/auth/login',
74
+ authCredentials: { email: 'test@example.com', password: 'secret123' },
75
+ authTokenPath: 'data.access_token',
76
+ authStorageKey: 'accessToken',
77
+ };
78
+ ```
79
+
80
+ If `authToken` is already set (via config, env var, or CLI), the auto-login step is skipped.
81
+
82
+ ## Clearing State
83
+
84
+ Each test runs in a fresh browser context (auto-clean). Use `clear_cookies` to explicitly clear cookies + localStorage + sessionStorage mid-test.
85
+
86
+ ## Reusable Auth Modules
87
+
88
+ Create `e2e/modules/login.json` or `e2e/modules/auth-token.json` and reference via:
89
+ ```json
90
+ { "$use": "login", "params": { "email": "...", "password": "..." } }
91
+ ```
@@ -0,0 +1,59 @@
1
+ # GraphQL Action Reference
2
+
3
+ The `gql` action executes GraphQL queries and mutations via browser `fetch`, with automatic auth token injection from `localStorage`. It also installs a `window.__e2eGql(query, vars)` helper for use in subsequent `evaluate` actions.
4
+
5
+ ## Action Fields
6
+
7
+ | Action | Fields | Behavior |
8
+ |--------|--------|----------|
9
+ | `gql` | `value` (query string, required), `text` (variables JSON, optional), `selector` (assertion JS expression, optional) | Sends a GraphQL request to the configured endpoint. Reads auth token from localStorage. Throws on GraphQL errors. Returns `{ value: response.data }`. Stores full response on `window.__e2eLastGql`. |
10
+
11
+ ## Config Fields
12
+
13
+ | Field | Default | Env Var | CLI Flag |
14
+ |-------|---------|---------|----------|
15
+ | `gqlEndpoint` | `/api/graphql` | `GQL_ENDPOINT` | `--gql-endpoint` |
16
+ | `gqlAuthHeader` | `Authorization` | `GQL_AUTH_HEADER` | `--gql-auth-header` |
17
+ | `gqlAuthKey` | `accessToken` | `GQL_AUTH_KEY` | `--gql-auth-key` |
18
+ | `gqlAuthPrefix` | `Bearer ` | `GQL_AUTH_PREFIX` | `--gql-auth-prefix` |
19
+
20
+ The endpoint path is appended to `location.origin`. The auth token is read from `localStorage[gqlAuthKey]` and sent as `gqlAuthHeader: gqlAuthPrefix + token`.
21
+
22
+ ## Examples
23
+
24
+ ### Basic query
25
+ ```json
26
+ { "type": "gql", "value": "{ users { id name } }" }
27
+ ```
28
+
29
+ ### With variables
30
+ ```json
31
+ { "type": "gql", "value": "query($id: ID, $orgId: ID) { orders(userId: $id, orgId: $orgId) { orderId status } }", "text": "{\"id\": \"abc-123\", \"orgId\": \"org-456\"}" }
32
+ ```
33
+
34
+ ### With inline assertion (selector field)
35
+ ```json
36
+ // selector is a JS expression where `r` is the full GraphQL response
37
+ { "type": "gql", "value": "{ pendingOrders(userId: \"abc-123\") { status } }", "selector": "r.data.pendingOrders.some(o => o.status === 'CANCELLED') ? 'FAIL: cancelled order found in pending list' : 'OK: all pending'" }
38
+ ```
39
+
40
+ ### Using the installed helper in evaluate
41
+ ```json
42
+ // After any gql action runs, window.__e2eGql(query, vars) is available
43
+ { "type": "gql", "value": "{ __typename }" }
44
+ { "type": "evaluate", "value": "(async () => { const r = await window.__e2eGql('query { orders(status: [PROCESSING]) { orderId } }'); for (const o of r.data.orders) await window.__e2eGql('mutation($id: ID, $input: OrderInput) { updateOrder(orderId: $id, input: $input) { orderId } }', { id: o.orderId, input: { status: 'COMPLETED' } }); return 'Updated ' + r.data.orders.length; })()" }
45
+ ```
46
+
47
+ ## Custom Auth Header Config
48
+
49
+ If your API uses a non-standard auth header (e.g., `x-api-key` instead of `Authorization`):
50
+
51
+ ```js
52
+ // e2e.config.js
53
+ export default {
54
+ gqlEndpoint: '/api/graphql',
55
+ gqlAuthHeader: 'x-api-key', // custom header name
56
+ gqlAuthKey: 'apiToken', // localStorage key to read from
57
+ gqlAuthPrefix: '', // no 'Bearer ' prefix — raw token
58
+ };
59
+ ```
@@ -0,0 +1,59 @@
1
+ # Issue-to-Test Verification Reference
2
+
3
+ Turns bug reports and feature requests into executable E2E tests.
4
+
5
+ ## Supported Providers
6
+
7
+ - **GitHub** (`github.com`) — requires `gh` CLI (`gh auth login`)
8
+ - **GitLab** (including self-hosted) — requires `glab` CLI (`glab auth login`)
9
+
10
+ Auto-detected from URL. All external commands use `execFileSync` (no shell injection).
11
+
12
+ ## Two AI Modes
13
+
14
+ ### 1. Prompt Mode (default, no API key)
15
+
16
+ `e2e_issue` MCP tool returns issue details + a structured prompt. Claude Code then uses `e2e_create_test` to create tests and `e2e_run` to execute them.
17
+
18
+ ### 2. Verify Mode (requires `ANTHROPIC_API_KEY`)
19
+
20
+ Calls Claude API directly to generate tests, runs them, and reports whether the bug is confirmed or not reproducible.
21
+
22
+ ## Config Fields
23
+
24
+ | Field | Description | Env Var |
25
+ |-------|-------------|---------|
26
+ | `anthropicApiKey` | Required for verify/generate mode | `ANTHROPIC_API_KEY` |
27
+ | `anthropicModel` | Claude model for generation (default: `claude-sonnet-4-5-20250929`) | `ANTHROPIC_MODEL` |
28
+
29
+ ## Bug Verification Logic
30
+
31
+ Generated tests assert **correct** behavior:
32
+ - **Test failure** = bug confirmed (expected behavior doesn't match reality)
33
+ - **All tests pass** = not reproducible
34
+
35
+ ## Test Categories (`testType`)
36
+
37
+ The `--test-type` CLI flag (or `testType` MCP parameter) controls what kind of tests the AI generates:
38
+
39
+ - **`e2e`** (default): UI-driven tests — navigate pages, interact with elements (click, type, select), verify visible state. Never uses `evaluate` for API calls.
40
+ - **`api`**: Backend API tests — use `evaluate` actions for GraphQL/REST calls, assert response shapes and values. No UI interaction needed.
41
+
42
+ CLI: `e2e-runner issue <url> --generate --test-type api`
43
+ MCP: `e2e_issue({ url, testType: "api" })`
44
+
45
+ ## GitLab Limitations
46
+
47
+ - Requires `glab` CLI installed and authenticated (`glab auth login`)
48
+ - Self-hosted GitLab instances supported via URL detection
49
+ - Verify mode works with GitLab issues but does NOT post comments back to the issue
50
+ - Private repos require `glab` to be authenticated with appropriate access
51
+ - The `authToken`/`authStorageKey` params on `e2e_issue` are for the **app under test**, not for GitLab API auth
52
+
53
+ ## Key Files
54
+
55
+ | File | Purpose |
56
+ |------|---------|
57
+ | `src/issues.js` | GitHub/GitLab provider drivers |
58
+ | `src/ai-generate.js` | AI prompt builder + Claude API |
59
+ | `src/verify.js` | Verification orchestrator: fetch + generate + run |
@@ -0,0 +1,60 @@
1
+ # Multi-Pool Support
2
+
3
+ A single runner can distribute tests across multiple Chrome pools on different machines. `src/pool-manager.js` abstracts pool selection behind a **least-pressure** strategy. `poolUrl` (singular string) remains fully backwards compatible.
4
+
5
+ ## Configuration
6
+
7
+ ```js
8
+ // Single pool (unchanged)
9
+ export default { poolUrl: 'ws://localhost:3333' };
10
+
11
+ // Multi-pool — tests distribute across all pools
12
+ export default {
13
+ poolUrls: ['ws://machine1:3333', 'ws://machine2:3333'],
14
+ concurrency: 15,
15
+ };
16
+
17
+ // CI environment profile
18
+ export default {
19
+ poolUrl: 'ws://localhost:3333',
20
+ environments: {
21
+ ci: { poolUrls: ['ws://ci-chrome1:3333', 'ws://ci-chrome2:3333'], concurrency: 10 },
22
+ },
23
+ };
24
+ ```
25
+
26
+ **CLI:** `e2e-runner run --all --pool-urls ws://m1:3333,ws://m2:3333`
27
+ **Env:** `CHROME_POOL_URLS=ws://m1:3333,ws://m2:3333` (comma-separated)
28
+
29
+ ## Pool Selection Algorithm (`selectPool`)
30
+
31
+ 1. Query all pools' `/pressure` in parallel
32
+ 2. Filter to reachable pools with `running < maxConcurrent`
33
+ 3. Sort by: lowest `running/maxConcurrent` → fewest queued → most free slots
34
+ 4. Return best candidate URL
35
+ 5. If all full, poll every 2s up to 60s, then pick least-pressured anyway
36
+
37
+ ## Pool-Aware Queue
38
+
39
+ Before opening a browser connection, each worker checks the pool's `/pressure` endpoint. If the pool is at capacity, the worker waits (polling every 2s, up to 60s) for a free slot instead of piling requests into browserless's internal queue. This prevents memory pressure and SIGKILL of Chrome processes under heavy load.
40
+
41
+ ## Failure Resilience
42
+
43
+ Dead pools are excluded from selection. `waitForAnyPool` succeeds if any pool responds. Failed connections trigger re-selection on the next worker.
44
+
45
+ ## Config Normalization in `loadConfig()`
46
+
47
+ - If `poolUrls` array is set → `config._poolUrls = poolUrls`, `config.poolUrl = poolUrls[0]`
48
+ - Else → `config._poolUrls = [config.poolUrl]`
49
+ - `config.poolUrls` is deleted after normalization (use `config._poolUrls` internally)
50
+
51
+ ## Key Functions (`src/pool-manager.js`)
52
+
53
+ | Function | Purpose |
54
+ |----------|---------|
55
+ | `getPoolUrls(config)` | Returns `config._poolUrls` (always an array) |
56
+ | `getAllPoolStatuses(poolUrls)` | Fetches `/pressure` from all pools in parallel |
57
+ | `getAggregatedPoolStatus(poolUrls)` | Combined view: `totalRunning`, `totalMaxConcurrent`, per-pool details |
58
+ | `waitForAnyPool(poolUrls)` | Blocks until at least one pool is reachable + available |
59
+ | `selectPool(poolUrls)` | Picks pool with lowest pressure ratio |
60
+ | `selectAndConnect(config)` | Convenience: selectPool + connectToPool in one call |
@@ -0,0 +1,62 @@
1
+ # Network Debugging Reference
2
+
3
+ ## Network Error Handling
4
+
5
+ ### `assert_no_network_errors` action
6
+
7
+ Checks accumulated `requestfailed` events during the test. If any network errors exist (e.g., `net::ERR_CONNECTION_REFUSED`), the test fails with details of each error URL. Place after critical page loads.
8
+
9
+ ### `failOnNetworkError` config option
10
+
11
+ When `true`, automatically fails any test that has network errors after all actions complete. Default: `false` (opt-in).
12
+
13
+ Set via: config `failOnNetworkError: true` | CLI `--fail-on-network-error` | env `FAIL_ON_NETWORK_ERROR=true` | MCP `failOnNetworkError: true`
14
+
15
+ ### `networkIgnoreDomains` config option
16
+
17
+ Array of domain substrings to filter from network error tracking. Errors from matching URLs are silently dropped by both `assert_no_network_errors` and `failOnNetworkError`.
18
+
19
+ Set via: config `networkIgnoreDomains: ['google-analytics.com', 'fonts.googleapis.com']` | CLI `--network-ignore-domains ga.com,fonts.com` | env `NETWORK_IGNORE_DOMAINS=ga.com,fonts.com` (comma-separated)
20
+
21
+ ## Network Request/Response Logging
22
+
23
+ All XHR/fetch requests are captured with full detail regardless of status code:
24
+
25
+ - `url`, `method`, `status`, `statusText`, `duration`
26
+ - `requestHeaders` — all request headers as object
27
+ - `requestBody` — POST body (from `req.postData()`)
28
+ - `responseHeaders` — all response headers as object
29
+ - `responseBody` — full response text (truncated at 50KB)
30
+
31
+ Response bodies are read asynchronously and flushed via `Promise.allSettled` before the browser disconnects. Data is stored in the `network_logs` column in SQLite and displayed in the dashboard.
32
+
33
+ ## MCP Response Optimization
34
+
35
+ The `e2e_run` MCP tool returns a compact `networkSummary` instead of full logs (~5KB vs ~400KB):
36
+
37
+ ```json
38
+ {
39
+ "networkSummary": [{
40
+ "name": "test-name",
41
+ "totalRequests": 37,
42
+ "statusDistribution": { "2xx": 30, "3xx": 5, "4xx": 1, "5xx": 0, "other": 1 },
43
+ "avgDurationMs": 245,
44
+ "failedRequests": [{ "url": "/api/x", "method": "POST", "status": 500 }],
45
+ "slowestRequests": [{ "url": "/api/y", "method": "GET", "status": 200, "duration": 1200 }]
46
+ }]
47
+ }
48
+ ```
49
+
50
+ ## Drill-Down Pattern
51
+
52
+ The response includes `runDbId` — the SQLite row ID. Use it with `e2e_network_logs` to drill down:
53
+
54
+ ```
55
+ 1. e2e_run → compact summary + runDbId
56
+ 2. e2e_network_logs(runDbId) → all requests (url, method, status, duration)
57
+ 3. e2e_network_logs(runDbId, errorsOnly: true) → only failed requests
58
+ 4. e2e_network_logs(runDbId, includeHeaders: true) → with headers
59
+ 5. e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
60
+ ```
61
+
62
+ Dashboard REST equivalent: `GET /api/db/runs/:id/network-logs?testName=X&errorsOnly=true&includeHeaders=true`
@@ -150,6 +150,10 @@ export default {
150
150
 
151
151
  Activate with `--env staging` or `E2E_ENV=staging`. Profile values override all other config.
152
152
 
153
+ ## CI Output Formats
154
+
155
+ Use `--output <format>` to control report output: `json` (default), `junit` (JUnit XML), or `both`. JUnit XML is generated without external dependencies and saved to `{screenshotsDir}/junit.xml`. The `generateJUnitXML()` function is also exported from the programmatic API.
156
+
153
157
  ## Config Priority (ascending)
154
158
 
155
159
  1. Hardcoded defaults