@matware/e2e-runner 1.2.1 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/.claude-plugin/marketplace.json +21 -0
  2. package/.mcp.json +2 -2
  3. package/.opencode/commands/create-test.md +63 -0
  4. package/.opencode/commands/run.md +50 -0
  5. package/.opencode/commands/verify-issue.md +62 -0
  6. package/.opencode/skills/e2e-testing/SKILL.md +181 -0
  7. package/.opencode/skills/e2e-testing/references/action-types.md +143 -0
  8. package/.opencode/skills/e2e-testing/references/auth-strategies.md +91 -0
  9. package/.opencode/skills/e2e-testing/references/graphql.md +59 -0
  10. package/.opencode/skills/e2e-testing/references/issue-verification.md +59 -0
  11. package/.opencode/skills/e2e-testing/references/multi-pool.md +60 -0
  12. package/.opencode/skills/e2e-testing/references/network-debugging.md +62 -0
  13. package/.opencode/skills/e2e-testing/references/test-json-format.md +163 -0
  14. package/.opencode/skills/e2e-testing/references/troubleshooting.md +224 -0
  15. package/.opencode/skills/e2e-testing/references/variables.md +41 -0
  16. package/.opencode/skills/e2e-testing/references/visual-verification.md +89 -0
  17. package/OPENCODE.md +166 -0
  18. package/README.md +581 -55
  19. package/agents/test-creator.md +54 -1
  20. package/agents/test-improver.md +37 -0
  21. package/bin/cli.js +408 -16
  22. package/commands/create-test.md +16 -1
  23. package/opencode.json +11 -0
  24. package/package.json +7 -2
  25. package/scripts/setup-opencode.sh +113 -0
  26. package/skills/e2e-testing/SKILL.md +10 -3
  27. package/skills/e2e-testing/references/action-types.md +48 -5
  28. package/skills/e2e-testing/references/auth-strategies.md +91 -0
  29. package/skills/e2e-testing/references/graphql.md +59 -0
  30. package/skills/e2e-testing/references/issue-verification.md +59 -0
  31. package/skills/e2e-testing/references/multi-pool.md +60 -0
  32. package/skills/e2e-testing/references/network-debugging.md +62 -0
  33. package/skills/e2e-testing/references/test-json-format.md +4 -0
  34. package/skills/e2e-testing/references/troubleshooting.md +44 -2
  35. package/skills/e2e-testing/references/variables.md +41 -0
  36. package/skills/e2e-testing/references/visual-verification.md +89 -0
  37. package/src/actions.js +324 -2
  38. package/src/ai-generate.js +58 -8
  39. package/src/config.js +143 -0
  40. package/src/dashboard.js +145 -13
  41. package/src/db.js +130 -2
  42. package/src/index.js +7 -6
  43. package/src/learner-sqlite.js +304 -0
  44. package/src/learner.js +8 -3
  45. package/src/mcp-tools.js +1121 -43
  46. package/src/module-resolver.js +37 -0
  47. package/src/narrate.js +37 -0
  48. package/src/pool-manager.js +223 -0
  49. package/src/reporter.js +82 -1
  50. package/src/runner.js +157 -28
  51. package/src/sync/auth.js +354 -0
  52. package/src/sync/client.js +572 -0
  53. package/src/sync/hub-routes.js +816 -0
  54. package/src/sync/index.js +68 -0
  55. package/src/sync/middleware.js +347 -0
  56. package/src/sync/queue.js +209 -0
  57. package/src/sync/schema.js +540 -0
  58. package/src/verify.js +10 -7
  59. package/src/watch.js +384 -0
  60. package/templates/build-dashboard.js +47 -6
  61. package/templates/dashboard/js/api.js +60 -0
  62. package/templates/dashboard/js/init.js +13 -0
  63. package/templates/dashboard/js/keyboard.js +46 -0
  64. package/templates/dashboard/js/state.js +40 -0
  65. package/templates/dashboard/js/toast.js +41 -0
  66. package/templates/dashboard/js/utils.js +196 -0
  67. package/templates/dashboard/js/view-live.js +143 -0
  68. package/templates/dashboard/js/view-runs.js +572 -0
  69. package/templates/dashboard/js/view-tests.js +294 -0
  70. package/templates/dashboard/js/view-watch.js +242 -0
  71. package/templates/dashboard/js/websocket.js +110 -0
  72. package/templates/dashboard/styles/base.css +69 -0
  73. package/templates/dashboard/styles/components.css +110 -0
  74. package/templates/dashboard/styles/view-live.css +74 -0
  75. package/templates/dashboard/styles/view-runs.css +207 -0
  76. package/templates/dashboard/styles/view-tests.css +96 -0
  77. package/templates/dashboard/styles/view-watch.css +53 -0
  78. package/templates/dashboard/template.html +165 -99
  79. package/templates/dashboard.html +1596 -541
  80. package/templates/sample-test.json +0 -8
  81. package/templates/dashboard/app.js +0 -1152
  82. package/templates/dashboard/styles.css +0 -413
@@ -0,0 +1,59 @@
1
+ # GraphQL Action Reference
2
+
3
+ The `gql` action executes GraphQL queries and mutations via browser `fetch`, with automatic auth token injection from `localStorage`. It also installs a `window.__e2eGql(query, vars)` helper for use in subsequent `evaluate` actions.
4
+
5
+ ## Action Fields
6
+
7
+ | Action | Fields | Behavior |
8
+ |--------|--------|----------|
9
+ | `gql` | `value` (query string, required), `text` (variables JSON, optional), `selector` (assertion JS expression, optional) | Sends a GraphQL request to the configured endpoint. Reads auth token from localStorage. Throws on GraphQL errors. Returns `{ value: response.data }`. Stores full response on `window.__e2eLastGql`. |
10
+
11
+ ## Config Fields
12
+
13
+ | Field | Default | Env Var | CLI Flag |
14
+ |-------|---------|---------|----------|
15
+ | `gqlEndpoint` | `/api/graphql` | `GQL_ENDPOINT` | `--gql-endpoint` |
16
+ | `gqlAuthHeader` | `Authorization` | `GQL_AUTH_HEADER` | `--gql-auth-header` |
17
+ | `gqlAuthKey` | `accessToken` | `GQL_AUTH_KEY` | `--gql-auth-key` |
18
+ | `gqlAuthPrefix` | `Bearer ` | `GQL_AUTH_PREFIX` | `--gql-auth-prefix` |
19
+
20
+ The endpoint path is appended to `location.origin`. The auth token is read from `localStorage[gqlAuthKey]` and sent as `gqlAuthHeader: gqlAuthPrefix + token`.
21
+
22
+ ## Examples
23
+
24
+ ### Basic query
25
+ ```json
26
+ { "type": "gql", "value": "{ users { id name } }" }
27
+ ```
28
+
29
+ ### With variables
30
+ ```json
31
+ { "type": "gql", "value": "query($id: ID, $orgId: ID) { orders(userId: $id, orgId: $orgId) { orderId status } }", "text": "{\"id\": \"abc-123\", \"orgId\": \"org-456\"}" }
32
+ ```
33
+
34
+ ### With inline assertion (selector field)
35
+ ```json
36
+ // selector is a JS expression where `r` is the full GraphQL response
37
+ { "type": "gql", "value": "{ pendingOrders(userId: \"abc-123\") { status } }", "selector": "r.data.pendingOrders.some(o => o.status === 'CANCELLED') ? 'FAIL: cancelled order found in pending list' : 'OK: all pending'" }
38
+ ```
39
+
40
+ ### Using the installed helper in evaluate
41
+ ```json
42
+ // After any gql action runs, window.__e2eGql(query, vars) is available
43
+ { "type": "gql", "value": "{ __typename }" }
44
+ { "type": "evaluate", "value": "(async () => { const r = await window.__e2eGql('query { orders(status: [PROCESSING]) { orderId } }'); for (const o of r.data.orders) await window.__e2eGql('mutation($id: ID, $input: OrderInput) { updateOrder(orderId: $id, input: $input) { orderId } }', { id: o.orderId, input: { status: 'COMPLETED' } }); return 'Updated ' + r.data.orders.length; })()" }
45
+ ```
46
+
47
+ ## Custom Auth Header Config
48
+
49
+ If your API uses a non-standard auth header (e.g., `x-api-key` instead of `Authorization`):
50
+
51
+ ```js
52
+ // e2e.config.js
53
+ export default {
54
+ gqlEndpoint: '/api/graphql',
55
+ gqlAuthHeader: 'x-api-key', // custom header name
56
+ gqlAuthKey: 'apiToken', // localStorage key to read from
57
+ gqlAuthPrefix: '', // no 'Bearer ' prefix — raw token
58
+ };
59
+ ```
@@ -0,0 +1,59 @@
1
+ # Issue-to-Test Verification Reference
2
+
3
+ Turns bug reports and feature requests into executable E2E tests.
4
+
5
+ ## Supported Providers
6
+
7
+ - **GitHub** (`github.com`) — requires `gh` CLI (`gh auth login`)
8
+ - **GitLab** (including self-hosted) — requires `glab` CLI (`glab auth login`)
9
+
10
+ Auto-detected from URL. All external commands use `execFileSync` (no shell injection).
11
+
12
+ ## Two AI Modes
13
+
14
+ ### 1. Prompt Mode (default, no API key)
15
+
16
+ `e2e_issue` MCP tool returns issue details + a structured prompt. Claude Code then uses `e2e_create_test` to create tests and `e2e_run` to execute them.
17
+
18
+ ### 2. Verify Mode (requires `ANTHROPIC_API_KEY`)
19
+
20
+ Calls Claude API directly to generate tests, runs them, and reports whether the bug is confirmed or not reproducible.
21
+
22
+ ## Config Fields
23
+
24
+ | Field | Description | Env Var |
25
+ |-------|-------------|---------|
26
+ | `anthropicApiKey` | Required for verify/generate mode | `ANTHROPIC_API_KEY` |
27
+ | `anthropicModel` | Claude model for generation (default: `claude-sonnet-4-5-20250929`) | `ANTHROPIC_MODEL` |
28
+
29
+ ## Bug Verification Logic
30
+
31
+ Generated tests assert **correct** behavior:
32
+ - **Test failure** = bug confirmed (expected behavior doesn't match reality)
33
+ - **All tests pass** = not reproducible
34
+
35
+ ## Test Categories (`testType`)
36
+
37
+ The `--test-type` CLI flag (or `testType` MCP parameter) controls what kind of tests the AI generates:
38
+
39
+ - **`e2e`** (default): UI-driven tests — navigate pages, interact with elements (click, type, select), verify visible state. Never uses `evaluate` for API calls.
40
+ - **`api`**: Backend API tests — use `evaluate` actions for GraphQL/REST calls, assert response shapes and values. No UI interaction needed.
41
+
42
+ CLI: `e2e-runner issue <url> --generate --test-type api`
43
+ MCP: `e2e_issue({ url, testType: "api" })`
44
+
45
+ ## GitLab Limitations
46
+
47
+ - Requires `glab` CLI installed and authenticated (`glab auth login`)
48
+ - Self-hosted GitLab instances supported via URL detection
49
+ - Verify mode works with GitLab issues but does NOT post comments back to the issue
50
+ - Private repos require `glab` to be authenticated with appropriate access
51
+ - The `authToken`/`authStorageKey` params on `e2e_issue` are for the **app under test**, not for GitLab API auth
52
+
53
+ ## Key Files
54
+
55
+ | File | Purpose |
56
+ |------|---------|
57
+ | `src/issues.js` | GitHub/GitLab provider drivers |
58
+ | `src/ai-generate.js` | AI prompt builder + Claude API |
59
+ | `src/verify.js` | Verification orchestrator: fetch + generate + run |
@@ -0,0 +1,60 @@
1
+ # Multi-Pool Support
2
+
3
+ A single runner can distribute tests across multiple Chrome pools on different machines. `src/pool-manager.js` abstracts pool selection behind a **least-pressure** strategy. `poolUrl` (singular string) remains fully backwards compatible.
4
+
5
+ ## Configuration
6
+
7
+ ```js
8
+ // Single pool (unchanged)
9
+ export default { poolUrl: 'ws://localhost:3333' };
10
+
11
+ // Multi-pool — tests distribute across all pools
12
+ export default {
13
+ poolUrls: ['ws://machine1:3333', 'ws://machine2:3333'],
14
+ concurrency: 15,
15
+ };
16
+
17
+ // CI environment profile
18
+ export default {
19
+ poolUrl: 'ws://localhost:3333',
20
+ environments: {
21
+ ci: { poolUrls: ['ws://ci-chrome1:3333', 'ws://ci-chrome2:3333'], concurrency: 10 },
22
+ },
23
+ };
24
+ ```
25
+
26
+ **CLI:** `e2e-runner run --all --pool-urls ws://m1:3333,ws://m2:3333`
27
+ **Env:** `CHROME_POOL_URLS=ws://m1:3333,ws://m2:3333` (comma-separated)
28
+
29
+ ## Pool Selection Algorithm (`selectPool`)
30
+
31
+ 1. Query all pools' `/pressure` in parallel
32
+ 2. Filter to reachable pools with `running < maxConcurrent`
33
+ 3. Sort by: lowest `running/maxConcurrent` → fewest queued → most free slots
34
+ 4. Return best candidate URL
35
+ 5. If all full, poll every 2s up to 60s, then pick least-pressured anyway
36
+
37
+ ## Pool-Aware Queue
38
+
39
+ Before opening a browser connection, each worker checks the pool's `/pressure` endpoint. If the pool is at capacity, the worker waits (polling every 2s, up to 60s) for a free slot instead of piling requests into browserless's internal queue. This prevents memory pressure and SIGKILL of Chrome processes under heavy load.
40
+
41
+ ## Failure Resilience
42
+
43
+ Dead pools are excluded from selection. `waitForAnyPool` succeeds if any pool responds. Failed connections trigger re-selection on the next worker.
44
+
45
+ ## Config Normalization in `loadConfig()`
46
+
47
+ - If `poolUrls` array is set → `config._poolUrls = poolUrls`, `config.poolUrl = poolUrls[0]`
48
+ - Else → `config._poolUrls = [config.poolUrl]`
49
+ - `config.poolUrls` is deleted after normalization (use `config._poolUrls` internally)
50
+
51
+ ## Key Functions (`src/pool-manager.js`)
52
+
53
+ | Function | Purpose |
54
+ |----------|---------|
55
+ | `getPoolUrls(config)` | Returns `config._poolUrls` (always an array) |
56
+ | `getAllPoolStatuses(poolUrls)` | Fetches `/pressure` from all pools in parallel |
57
+ | `getAggregatedPoolStatus(poolUrls)` | Combined view: `totalRunning`, `totalMaxConcurrent`, per-pool details |
58
+ | `waitForAnyPool(poolUrls)` | Blocks until at least one pool is reachable + available |
59
+ | `selectPool(poolUrls)` | Picks pool with lowest pressure ratio |
60
+ | `selectAndConnect(config)` | Convenience: selectPool + connectToPool in one call |
@@ -0,0 +1,62 @@
1
+ # Network Debugging Reference
2
+
3
+ ## Network Error Handling
4
+
5
+ ### `assert_no_network_errors` action
6
+
7
+ Checks accumulated `requestfailed` events during the test. If any network errors exist (e.g., `net::ERR_CONNECTION_REFUSED`), the test fails with details of each error URL. Place after critical page loads.
8
+
9
+ ### `failOnNetworkError` config option
10
+
11
+ When `true`, automatically fails any test that has network errors after all actions complete. Default: `false` (opt-in).
12
+
13
+ Set via: config `failOnNetworkError: true` | CLI `--fail-on-network-error` | env `FAIL_ON_NETWORK_ERROR=true` | MCP `failOnNetworkError: true`
14
+
15
+ ### `networkIgnoreDomains` config option
16
+
17
+ Array of domain substrings to filter from network error tracking. Errors from matching URLs are silently dropped by both `assert_no_network_errors` and `failOnNetworkError`.
18
+
19
+ Set via: config `networkIgnoreDomains: ['google-analytics.com', 'fonts.googleapis.com']` | CLI `--network-ignore-domains ga.com,fonts.com` | env `NETWORK_IGNORE_DOMAINS=ga.com,fonts.com` (comma-separated)
20
+
21
+ ## Network Request/Response Logging
22
+
23
+ All XHR/fetch requests are captured with full detail regardless of status code:
24
+
25
+ - `url`, `method`, `status`, `statusText`, `duration`
26
+ - `requestHeaders` — all request headers as object
27
+ - `requestBody` — POST body (from `req.postData()`)
28
+ - `responseHeaders` — all response headers as object
29
+ - `responseBody` — full response text (truncated at 50KB)
30
+
31
+ Response bodies are read asynchronously and flushed via `Promise.allSettled` before the browser disconnects. Data is stored in the `network_logs` column in SQLite and displayed in the dashboard.
32
+
33
+ ## MCP Response Optimization
34
+
35
+ The `e2e_run` MCP tool returns a compact `networkSummary` instead of full logs (~5KB vs ~400KB):
36
+
37
+ ```json
38
+ {
39
+ "networkSummary": [{
40
+ "name": "test-name",
41
+ "totalRequests": 37,
42
+ "statusDistribution": { "2xx": 30, "3xx": 5, "4xx": 1, "5xx": 0, "other": 1 },
43
+ "avgDurationMs": 245,
44
+ "failedRequests": [{ "url": "/api/x", "method": "POST", "status": 500 }],
45
+ "slowestRequests": [{ "url": "/api/y", "method": "GET", "status": 200, "duration": 1200 }]
46
+ }]
47
+ }
48
+ ```
49
+
50
+ ## Drill-Down Pattern
51
+
52
+ The response includes `runDbId` — the SQLite row ID. Use it with `e2e_network_logs` to drill down:
53
+
54
+ ```
55
+ 1. e2e_run → compact summary + runDbId
56
+ 2. e2e_network_logs(runDbId) → all requests (url, method, status, duration)
57
+ 3. e2e_network_logs(runDbId, errorsOnly: true) → only failed requests
58
+ 4. e2e_network_logs(runDbId, includeHeaders: true) → with headers
59
+ 5. e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
60
+ ```
61
+
62
+ Dashboard REST equivalent: `GET /api/db/runs/:id/network-logs?testName=X&errorsOnly=true&includeHeaders=true`
@@ -0,0 +1,163 @@
1
+ # Test JSON Format Reference
2
+
3
+ ## Basic Format (Array)
4
+
5
+ A test file is a JSON array of test objects:
6
+
7
+ ```json
8
+ [
9
+ {
10
+ "name": "test-name",
11
+ "actions": [
12
+ { "type": "goto", "value": "/page" },
13
+ { "type": "assert_text", "text": "Expected content" }
14
+ ]
15
+ }
16
+ ]
17
+ ```
18
+
19
+ ## Object Format (with Hooks)
20
+
21
+ When hooks are needed, use the object format:
22
+
23
+ ```json
24
+ {
25
+ "hooks": {
26
+ "beforeAll": [{ "type": "goto", "value": "/setup" }],
27
+ "beforeEach": [{ "type": "goto", "value": "/" }],
28
+ "afterEach": [],
29
+ "afterAll": []
30
+ },
31
+ "tests": [
32
+ { "name": "test-1", "actions": [...] }
33
+ ]
34
+ }
35
+ ```
36
+
37
+ **Hook lifecycle:**
38
+ - `beforeAll` — runs once before all tests (on a separate browser page, state does NOT carry over)
39
+ - `beforeEach` — runs before each individual test (on the test's own page)
40
+ - `afterEach` — runs after each test
41
+ - `afterAll` — runs once after all tests
42
+
43
+ > **Warning**: `beforeAll` runs on a separate page that closes before tests start. Don't use it for browser state setup (cookies, localStorage). Use `beforeEach` instead.
44
+
45
+ ## Test Options
46
+
47
+ | Field | Type | Description |
48
+ |-------|------|-------------|
49
+ | `name` | string | **Required.** Test identifier. |
50
+ | `actions` | array | **Required.** Sequential browser actions. |
51
+ | `expect` | string | Visual verification description. Triggers auto-screenshot + AI judgment. |
52
+ | `serial` | boolean | Run sequentially after all parallel tests (for shared state). |
53
+ | `retries` | number | Per-test retry count on failure. Overrides global config. |
54
+ | `timeout` | number | Per-test timeout in ms. Overrides global `testTimeout` (default 60000). |
55
+
56
+ ## Serial Tests
57
+
58
+ Tests that share mutable state should be marked serial to prevent race conditions:
59
+
60
+ ```json
61
+ { "name": "create-record", "serial": true, "actions": [...] },
62
+ { "name": "verify-record", "serial": true, "actions": [...] }
63
+ ```
64
+
65
+ Serial tests run one-at-a-time **after** all parallel tests finish.
66
+
67
+ ## Retry Behavior
68
+
69
+ ### Test-level retries
70
+ ```json
71
+ { "name": "flaky-test", "retries": 3, "actions": [...] }
72
+ ```
73
+
74
+ Or globally: `--retries 2` / `retries: 2` in config. Each retry gets its own timeout. Flaky tests (pass after retry) are logged as "flaky".
75
+
76
+ ### Action-level retries
77
+ ```json
78
+ { "type": "click", "selector": "#dynamic-btn", "retries": 3 }
79
+ ```
80
+
81
+ Or globally: `--action-retries 2`. Delay between action retries: `actionRetryDelay` (default 500ms).
82
+
83
+ ## Reusable Modules
84
+
85
+ Create modules with `e2e_create_module`, reference them in tests:
86
+
87
+ ```json
88
+ {
89
+ "name": "login-test",
90
+ "actions": [
91
+ { "$use": "auth-login", "params": { "email": "admin@test.com", "password": "secret" } },
92
+ { "type": "assert_url", "value": "/dashboard" }
93
+ ]
94
+ }
95
+ ```
96
+
97
+ Module definition (in `e2e/modules/auth-login.json`):
98
+ ```json
99
+ {
100
+ "$module": "auth-login",
101
+ "description": "Log in with email/password",
102
+ "params": {
103
+ "email": { "required": true, "description": "User email" },
104
+ "password": { "required": true, "description": "User password" }
105
+ },
106
+ "actions": [
107
+ { "type": "goto", "value": "/login" },
108
+ { "type": "type", "selector": "#email", "value": "{{email}}" },
109
+ { "type": "type", "selector": "#password", "value": "{{password}}" },
110
+ { "type": "click", "text": "Sign In" },
111
+ { "type": "wait", "selector": ".dashboard" }
112
+ ]
113
+ }
114
+ ```
115
+
116
+ ## Suite Naming & Ordering
117
+
118
+ Files can have numeric prefixes for execution order:
119
+ - `01-auth.json`, `02-dashboard.json`, `03-settings.json`
120
+
121
+ The `--suite` flag strips the prefix when matching: `--suite auth` finds `01-auth.json`.
122
+
123
+ ## Excluding Tests
124
+
125
+ Use `exclude` in config to skip files when running `--all`:
126
+
127
+ ```js
128
+ // e2e.config.js
129
+ export default {
130
+ exclude: ['explore-*', 'debug-*', 'draft-*']
131
+ };
132
+ ```
133
+
134
+ Individual `--suite` runs are not affected by exclude patterns.
135
+
136
+ ## Environment Profiles
137
+
138
+ Define named profiles in config:
139
+
140
+ ```js
141
+ // e2e.config.js
142
+ export default {
143
+ baseUrl: 'http://host.docker.internal:3000',
144
+ environments: {
145
+ staging: { baseUrl: 'https://staging.example.com' },
146
+ production: { baseUrl: 'https://example.com', concurrency: 5 }
147
+ }
148
+ };
149
+ ```
150
+
151
+ Activate with `--env staging` or `E2E_ENV=staging`. Profile values override all other config.
152
+
153
+ ## CI Output Formats
154
+
155
+ Use `--output <format>` to control report output: `json` (default), `junit` (JUnit XML), or `both`. JUnit XML is generated without external dependencies and saved to `{screenshotsDir}/junit.xml`. The `generateJUnitXML()` function is also exported from the programmatic API.
156
+
157
+ ## Config Priority (ascending)
158
+
159
+ 1. Hardcoded defaults
160
+ 2. `e2e.config.js` or `e2e.config.json`
161
+ 3. Environment variables (`BASE_URL`, `CONCURRENCY`, etc.)
162
+ 4. CLI flags (`--base-url`, `--concurrency`, etc.)
163
+ 5. Environment profile merge (via `--env`)
@@ -0,0 +1,224 @@
1
+ # Troubleshooting Guide
2
+
3
+ ## Pool Connection Issues
4
+
5
+ ### "Pool not reachable" / Connection refused
6
+
7
+ **Cause**: Chrome pool (browserless/chrome Docker container) is not running.
8
+
9
+ **Fix**:
10
+ ```bash
11
+ npx e2e-runner pool start
12
+ npx e2e-runner pool status # verify it's running
13
+ ```
14
+
15
+ Pool management is CLI-only — `pool start` and `pool stop` are not available via MCP.
16
+
17
+ ### "Pool at capacity" / Tests queuing
18
+
19
+ **Cause**: All Chrome sessions are occupied.
20
+
21
+ **Fix**: Increase capacity or reduce concurrency:
22
+ ```bash
23
+ npx e2e-runner pool stop
24
+ npx e2e-runner pool start --max-sessions 10
25
+ ```
26
+ Or reduce test concurrency: `--concurrency 2`
27
+
28
+ The runner checks `/pressure` before each connection and waits up to 60s for a free slot.
29
+
30
+ ### Docker not running
31
+
32
+ **Cause**: Docker daemon is not started.
33
+
34
+ **Fix**: Start Docker Desktop or `sudo systemctl start docker`, then `npx e2e-runner pool start`.
35
+
36
+ ## React / SPA Issues
37
+
38
+ ### React inputs not updating state
39
+
40
+ **Symptom**: `type` action enters text but React state doesn't change (form validation fails, submit disabled).
41
+
42
+ **Fix**: Use `type_react` instead of `type` for React controlled inputs:
43
+ ```json
44
+ { "type": "type_react", "selector": "#email", "value": "user@test.com" }
45
+ ```
46
+
47
+ `type_react` uses the native value setter and dispatches `input` + `change` events that React's synthetic event system recognizes.
48
+
49
+ ### SPA navigation not completing
50
+
51
+ **Symptom**: `goto` hangs or times out on client-side route changes.
52
+
53
+ **Fix**: Use `navigate` instead of `goto` for SPA route changes:
54
+ ```json
55
+ { "type": "navigate", "value": "/new-page" }
56
+ ```
57
+
58
+ `navigate` uses a 5s race timeout and won't block if `load` doesn't fire (common in SPAs).
59
+
60
+ ### MUI autocomplete not opening
61
+
62
+ **Symptom**: Clicking or typing in an MUI Autocomplete doesn't open the dropdown.
63
+
64
+ **Fix**: Use `focus_autocomplete` to properly focus by label text:
65
+ ```json
66
+ { "type": "focus_autocomplete", "text": "Search by name" },
67
+ { "type": "type_react", "selector": "#autocomplete-input", "value": "search term" },
68
+ { "type": "click_option", "text": "Desired option" }
69
+ ```
70
+
71
+ ## Flaky Tests
72
+
73
+ ### Intermittent failures on dynamic content
74
+
75
+ **Symptom**: Tests pass sometimes, fail others. Usually timing-related.
76
+
77
+ **Fixes**:
78
+ 1. Add explicit `wait` before assertions:
79
+ ```json
80
+ { "type": "wait", "selector": ".data-loaded" },
81
+ { "type": "assert_text", "text": "Expected content" }
82
+ ```
83
+
84
+ 2. Use action-level retries for known flaky selectors:
85
+ ```json
86
+ { "type": "click", "selector": "#dynamic-btn", "retries": 3 }
87
+ ```
88
+
89
+ 3. Use test-level retries:
90
+ ```json
91
+ { "name": "flaky-test", "retries": 2, "actions": [...] }
92
+ ```
93
+
94
+ 4. Check the learning system for patterns:
95
+ ```
96
+ e2e_learnings("flaky") → identify consistently flaky tests
97
+ e2e_learnings("selectors") → find unstable selectors
98
+ ```
99
+
100
+ ### Tests interfering with each other
101
+
102
+ **Symptom**: Tests pass individually but fail when run together.
103
+
104
+ **Fix**: Mark tests that share mutable state as `serial`:
105
+ ```json
106
+ { "name": "create-item", "serial": true, "actions": [...] },
107
+ { "name": "verify-item", "serial": true, "actions": [...] }
108
+ ```
109
+
110
+ ## Timeout Issues
111
+
112
+ ### Test timeout (default 60s)
113
+
114
+ **Fix**: Increase per-test or globally:
115
+ ```json
116
+ { "name": "slow-test", "timeout": 120000, "actions": [...] }
117
+ ```
118
+ Or globally: `--test-timeout 120000`
119
+
120
+ ### Action timeout (default 10s)
121
+
122
+ Each action's `waitForSelector` uses the default timeout. Override per-action:
123
+ ```json
124
+ { "type": "wait", "selector": ".slow-element", "timeout": 30000 }
125
+ ```
126
+ Or globally: `--timeout 30000`
127
+
128
+ ## Network Errors
129
+
130
+ ### Tests passing but network requests failing
131
+
132
+ **Symptom**: Tests pass but `networkSummary` shows failed requests.
133
+
134
+ **Fix**: Enable strict mode to fail tests with network errors:
135
+ ```
136
+ e2e_run({ all: true, failOnNetworkError: true })
137
+ ```
138
+
139
+ Or use `assert_no_network_errors` at specific points:
140
+ ```json
141
+ { "type": "goto", "value": "/api-heavy-page" },
142
+ { "type": "wait", "selector": ".loaded" },
143
+ { "type": "assert_no_network_errors" }
144
+ ```
145
+
146
+ ### Investigating specific failures
147
+
148
+ Use network log drill-down:
149
+ ```
150
+ e2e_network_logs(runDbId, errorsOnly: true) → see all failed requests
151
+ e2e_network_logs(runDbId, urlPattern: "/api/users") → filter by URL
152
+ e2e_network_logs(runDbId, testName: "create-user", includeBodies: true) → full request/response
153
+ ```
154
+
155
+ ## Common Mistakes
156
+
157
+ ### Using `beforeAll` for browser state
158
+
159
+ `beforeAll` runs on a separate page that closes before tests. Use `beforeEach` for state setup.
160
+
161
+ ### Using `evaluate` for simple assertions
162
+
163
+ Prefer granular assertion actions over `evaluate` with inline JS:
164
+ ```json
165
+ // Bad: verbose, error-prone
166
+ { "type": "evaluate", "value": "if (!document.querySelector('h1').textContent.includes('Dashboard')) throw 'not found'" }
167
+
168
+ // Good: clear, auto-waits
169
+ { "type": "assert_element_text", "selector": "h1", "text": "Dashboard" }
170
+ ```
171
+
172
+ ### Forgetting `cwd` in MCP calls
173
+
174
+ All MCP tools need `cwd` to resolve config files and test directories. Always pass the project root.
175
+
176
+ ### Path-only `assert_url`
177
+
178
+ When checking paths, use path-only format (starts with `/`):
179
+ ```json
180
+ { "type": "assert_url", "value": "/dashboard" }
181
+ ```
182
+ This compares against the pathname only, ignoring the `host.docker.internal` origin.
183
+
184
+ ## Action Type Pre-Validation
185
+
186
+ All action types are validated at **load time** (before any browser connections). If a test file contains an unknown action type (e.g., a typo like `"clik"`), loading throws immediately with the location:
187
+
188
+ ```
189
+ Unknown action type(s) in auth.json: "clik" in test "login-test"
190
+ ```
191
+
192
+ The `KNOWN_ACTION_TYPES` Set in `src/actions.js` is the single source of truth. Unknown actions also throw at runtime as a safety net.
193
+
194
+ ## Screenshot Hashes
195
+
196
+ Every screenshot captured during a run is assigned a short hash (`ss:a3f2b1c9`) — the first 8 hex chars of the SHA-256 of its file path. Hashes are deterministic and computed identically on the server (Node `crypto`) and in the browser (Web Crypto API).
197
+
198
+ **Flow**: screenshot saved on disk → `saveRun()` registers hash in SQLite `screenshot_hashes` table → dashboard shows `[ss:XXXXXXXX]` badge (click to copy) → user pastes hash in Claude Code → `e2e_screenshot` MCP tool looks up hash, reads file, returns the image.
199
+
200
+ - Hashes are registered inside the `saveRun()` transaction (covers action, error, verification, and baseline screenshots)
201
+ - The `ss:` prefix is optional when calling `e2e_screenshot` — stripped during lookup
202
+ - Dashboard computes hashes client-side (Web Crypto) for the Live view (before `persistRun()` writes to DB)
203
+ - Run detail API (`/api/db/runs/:id`) includes `screenshotHashes` map per test result
204
+ - Dashboard endpoint `/api/screenshot-hash/:hash` serves the image by hash
205
+ - Dashboard Screenshots view has a **search bar** — type a hash to find and display the screenshot
206
+
207
+ ## Web Dashboard
208
+
209
+ **`src/dashboard.js`** — HTTP server, REST API, WebSocket broadcast, pool polling.
210
+ **`templates/dashboard.html`** — SPA, dark theme, vanilla JS, safe DOM (textContent + createEl helper).
211
+
212
+ **Features:**
213
+ - Live test execution with WebSocket updates
214
+ - Run history with inline detail expansion
215
+ - Screenshots gallery with hash badges and hash search
216
+ - Network request logs with clickable expandable rows (full request/response detail)
217
+ - Pool status monitoring
218
+ - Multi-project support via project selector
219
+ - Variables tab with masked values, inline edit, add, and delete
220
+
221
+ **CLI:** `e2e-runner dashboard [--port 8484]`
222
+ **MCP tools:** `e2e_dashboard_start`, `e2e_dashboard_stop`
223
+
224
+ Config defaults: `dashboardPort: 8484`, `maxHistoryRuns: 100`
@@ -0,0 +1,41 @@
1
+ # Variables Reference
2
+
3
+ Variables replace hardcoded sensitive values (JWT tokens, user IDs, API keys, etc.) in test JSON. Stored in SQLite (`~/.e2e-runner/dashboard.db`), scoped per project and per suite, editable from the dashboard UI.
4
+
5
+ ## Syntax
6
+
7
+ ```
8
+ {{var.TOKEN}} → resolves from DB (suite scope → project scope)
9
+ {{env.MY_VAR}} → resolves from process.env
10
+ {{param}} → existing module param substitution (unchanged)
11
+ ```
12
+
13
+ **Resolution priority:** suite vars > project vars > error if not found.
14
+
15
+ ## Usage in Test JSON
16
+
17
+ ```json
18
+ { "$use": "auth-jwt", "params": { "token": "{{var.JWT_TOKEN}}", "orgId": "{{var.ORG_ID}}" } }
19
+ { "type": "goto", "value": "/users/{{var.USER_ID}}/profile" }
20
+ { "type": "gql", "value": "{ user(id: \"{{var.USER_ID}}\") { name } }" }
21
+ ```
22
+
23
+ ## MCP Tool (`e2e_vars`)
24
+
25
+ ```
26
+ e2e_vars({ action: "set", key: "TOKEN", value: "abc123", scope: "project" })
27
+ e2e_vars({ action: "set", key: "TOKEN", value: "xyz789", scope: "auth" }) // suite-specific override
28
+ e2e_vars({ action: "list" })
29
+ e2e_vars({ action: "get", key: "TOKEN" })
30
+ e2e_vars({ action: "delete", key: "TOKEN", scope: "project" })
31
+ ```
32
+
33
+ ## Dashboard UI
34
+
35
+ Variables tab shows all variables grouped by scope. Values are masked by default (click to reveal). Inline edit, add new, and delete are supported.
36
+
37
+ ## REST API
38
+
39
+ - `GET /api/db/projects/:id/variables` — list all vars for project
40
+ - `PUT /api/db/projects/:id/variables` — set a variable `{ scope, key, value }`
41
+ - `DELETE /api/db/projects/:id/variables/:scope/:key` — delete a variable