@matware/e2e-runner 1.2.1 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +21 -0
- package/.mcp.json +2 -2
- package/.opencode/commands/create-test.md +63 -0
- package/.opencode/commands/run.md +50 -0
- package/.opencode/commands/verify-issue.md +62 -0
- package/.opencode/skills/e2e-testing/SKILL.md +181 -0
- package/.opencode/skills/e2e-testing/references/action-types.md +143 -0
- package/.opencode/skills/e2e-testing/references/auth-strategies.md +91 -0
- package/.opencode/skills/e2e-testing/references/graphql.md +59 -0
- package/.opencode/skills/e2e-testing/references/issue-verification.md +59 -0
- package/.opencode/skills/e2e-testing/references/multi-pool.md +60 -0
- package/.opencode/skills/e2e-testing/references/network-debugging.md +62 -0
- package/.opencode/skills/e2e-testing/references/test-json-format.md +163 -0
- package/.opencode/skills/e2e-testing/references/troubleshooting.md +224 -0
- package/.opencode/skills/e2e-testing/references/variables.md +41 -0
- package/.opencode/skills/e2e-testing/references/visual-verification.md +89 -0
- package/OPENCODE.md +166 -0
- package/README.md +581 -55
- package/agents/test-creator.md +54 -1
- package/agents/test-improver.md +37 -0
- package/bin/cli.js +408 -16
- package/commands/create-test.md +16 -1
- package/opencode.json +11 -0
- package/package.json +7 -2
- package/scripts/setup-opencode.sh +113 -0
- package/skills/e2e-testing/SKILL.md +10 -3
- package/skills/e2e-testing/references/action-types.md +48 -5
- package/skills/e2e-testing/references/auth-strategies.md +91 -0
- package/skills/e2e-testing/references/graphql.md +59 -0
- package/skills/e2e-testing/references/issue-verification.md +59 -0
- package/skills/e2e-testing/references/multi-pool.md +60 -0
- package/skills/e2e-testing/references/network-debugging.md +62 -0
- package/skills/e2e-testing/references/test-json-format.md +4 -0
- package/skills/e2e-testing/references/troubleshooting.md +44 -2
- package/skills/e2e-testing/references/variables.md +41 -0
- package/skills/e2e-testing/references/visual-verification.md +89 -0
- package/src/actions.js +324 -2
- package/src/ai-generate.js +58 -8
- package/src/config.js +143 -0
- package/src/dashboard.js +145 -13
- package/src/db.js +130 -2
- package/src/index.js +7 -6
- package/src/learner-sqlite.js +304 -0
- package/src/learner.js +8 -3
- package/src/mcp-tools.js +1121 -43
- package/src/module-resolver.js +37 -0
- package/src/narrate.js +37 -0
- package/src/pool-manager.js +223 -0
- package/src/reporter.js +82 -1
- package/src/runner.js +157 -28
- package/src/sync/auth.js +354 -0
- package/src/sync/client.js +572 -0
- package/src/sync/hub-routes.js +816 -0
- package/src/sync/index.js +68 -0
- package/src/sync/middleware.js +347 -0
- package/src/sync/queue.js +209 -0
- package/src/sync/schema.js +540 -0
- package/src/verify.js +10 -7
- package/src/watch.js +384 -0
- package/templates/build-dashboard.js +47 -6
- package/templates/dashboard/js/api.js +60 -0
- package/templates/dashboard/js/init.js +13 -0
- package/templates/dashboard/js/keyboard.js +46 -0
- package/templates/dashboard/js/state.js +40 -0
- package/templates/dashboard/js/toast.js +41 -0
- package/templates/dashboard/js/utils.js +196 -0
- package/templates/dashboard/js/view-live.js +143 -0
- package/templates/dashboard/js/view-runs.js +572 -0
- package/templates/dashboard/js/view-tests.js +294 -0
- package/templates/dashboard/js/view-watch.js +242 -0
- package/templates/dashboard/js/websocket.js +110 -0
- package/templates/dashboard/styles/base.css +69 -0
- package/templates/dashboard/styles/components.css +110 -0
- package/templates/dashboard/styles/view-live.css +74 -0
- package/templates/dashboard/styles/view-runs.css +207 -0
- package/templates/dashboard/styles/view-tests.css +96 -0
- package/templates/dashboard/styles/view-watch.css +53 -0
- package/templates/dashboard/template.html +165 -99
- package/templates/dashboard.html +1596 -541
- package/templates/sample-test.json +0 -8
- package/templates/dashboard/app.js +0 -1152
- package/templates/dashboard/styles.css +0 -413
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# GraphQL Action Reference
|
|
2
|
+
|
|
3
|
+
The `gql` action executes GraphQL queries and mutations via browser `fetch`, with automatic auth token injection from `localStorage`. It also installs a `window.__e2eGql(query, vars)` helper for use in subsequent `evaluate` actions.
|
|
4
|
+
|
|
5
|
+
## Action Fields
|
|
6
|
+
|
|
7
|
+
| Action | Fields | Behavior |
|
|
8
|
+
|--------|--------|----------|
|
|
9
|
+
| `gql` | `value` (query string, required), `text` (variables JSON, optional), `selector` (assertion JS expression, optional) | Sends a GraphQL request to the configured endpoint. Reads auth token from localStorage. Throws on GraphQL errors. Returns `{ value: response.data }`. Stores full response on `window.__e2eLastGql`. |
|
|
10
|
+
|
|
11
|
+
## Config Fields
|
|
12
|
+
|
|
13
|
+
| Field | Default | Env Var | CLI Flag |
|
|
14
|
+
|-------|---------|---------|----------|
|
|
15
|
+
| `gqlEndpoint` | `/api/graphql` | `GQL_ENDPOINT` | `--gql-endpoint` |
|
|
16
|
+
| `gqlAuthHeader` | `Authorization` | `GQL_AUTH_HEADER` | `--gql-auth-header` |
|
|
17
|
+
| `gqlAuthKey` | `accessToken` | `GQL_AUTH_KEY` | `--gql-auth-key` |
|
|
18
|
+
| `gqlAuthPrefix` | `Bearer ` | `GQL_AUTH_PREFIX` | `--gql-auth-prefix` |
|
|
19
|
+
|
|
20
|
+
The endpoint path is appended to `location.origin`. The auth token is read from `localStorage[gqlAuthKey]` and sent as `gqlAuthHeader: gqlAuthPrefix + token`.
|
|
21
|
+
|
|
22
|
+
## Examples
|
|
23
|
+
|
|
24
|
+
### Basic query
|
|
25
|
+
```json
|
|
26
|
+
{ "type": "gql", "value": "{ users { id name } }" }
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
### With variables
|
|
30
|
+
```json
|
|
31
|
+
{ "type": "gql", "value": "query($id: ID, $orgId: ID) { orders(userId: $id, orgId: $orgId) { orderId status } }", "text": "{\"id\": \"abc-123\", \"orgId\": \"org-456\"}" }
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
### With inline assertion (selector field)
|
|
35
|
+
```json
|
|
36
|
+
// selector is a JS expression where `r` is the full GraphQL response
|
|
37
|
+
{ "type": "gql", "value": "{ pendingOrders(userId: \"abc-123\") { status } }", "selector": "r.data.pendingOrders.some(o => o.status === 'CANCELLED') ? 'FAIL: cancelled order found in pending list' : 'OK: all pending'" }
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### Using the installed helper in evaluate
|
|
41
|
+
```json
|
|
42
|
+
// After any gql action runs, window.__e2eGql(query, vars) is available
|
|
43
|
+
{ "type": "gql", "value": "{ __typename }" }
|
|
44
|
+
{ "type": "evaluate", "value": "(async () => { const r = await window.__e2eGql('query { orders(status: [PROCESSING]) { orderId } }'); for (const o of r.data.orders) await window.__e2eGql('mutation($id: ID, $input: OrderInput) { updateOrder(orderId: $id, input: $input) { orderId } }', { id: o.orderId, input: { status: 'COMPLETED' } }); return 'Updated ' + r.data.orders.length; })()" }
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Custom Auth Header Config
|
|
48
|
+
|
|
49
|
+
If your API uses a non-standard auth header (e.g., `x-api-key` instead of `Authorization`):
|
|
50
|
+
|
|
51
|
+
```js
|
|
52
|
+
// e2e.config.js
|
|
53
|
+
export default {
|
|
54
|
+
gqlEndpoint: '/api/graphql',
|
|
55
|
+
gqlAuthHeader: 'x-api-key', // custom header name
|
|
56
|
+
gqlAuthKey: 'apiToken', // localStorage key to read from
|
|
57
|
+
gqlAuthPrefix: '', // no 'Bearer ' prefix — raw token
|
|
58
|
+
};
|
|
59
|
+
```
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Issue-to-Test Verification Reference
|
|
2
|
+
|
|
3
|
+
Turns bug reports and feature requests into executable E2E tests.
|
|
4
|
+
|
|
5
|
+
## Supported Providers
|
|
6
|
+
|
|
7
|
+
- **GitHub** (`github.com`) — requires `gh` CLI (`gh auth login`)
|
|
8
|
+
- **GitLab** (including self-hosted) — requires `glab` CLI (`glab auth login`)
|
|
9
|
+
|
|
10
|
+
Auto-detected from URL. All external commands use `execFileSync` (no shell injection).
|
|
11
|
+
|
|
12
|
+
## Two AI Modes
|
|
13
|
+
|
|
14
|
+
### 1. Prompt Mode (default, no API key)
|
|
15
|
+
|
|
16
|
+
`e2e_issue` MCP tool returns issue details + a structured prompt. Claude Code then uses `e2e_create_test` to create tests and `e2e_run` to execute them.
|
|
17
|
+
|
|
18
|
+
### 2. Verify Mode (requires `ANTHROPIC_API_KEY`)
|
|
19
|
+
|
|
20
|
+
Calls Claude API directly to generate tests, runs them, and reports whether the bug is confirmed or not reproducible.
|
|
21
|
+
|
|
22
|
+
## Config Fields
|
|
23
|
+
|
|
24
|
+
| Field | Description | Env Var |
|
|
25
|
+
|-------|-------------|---------|
|
|
26
|
+
| `anthropicApiKey` | Required for verify/generate mode | `ANTHROPIC_API_KEY` |
|
|
27
|
+
| `anthropicModel` | Claude model for generation (default: `claude-sonnet-4-5-20250929`) | `ANTHROPIC_MODEL` |
|
|
28
|
+
|
|
29
|
+
## Bug Verification Logic
|
|
30
|
+
|
|
31
|
+
Generated tests assert **correct** behavior:
|
|
32
|
+
- **Test failure** = bug confirmed (expected behavior doesn't match reality)
|
|
33
|
+
- **All tests pass** = not reproducible
|
|
34
|
+
|
|
35
|
+
## Test Categories (`testType`)
|
|
36
|
+
|
|
37
|
+
The `--test-type` CLI flag (or `testType` MCP parameter) controls what kind of tests the AI generates:
|
|
38
|
+
|
|
39
|
+
- **`e2e`** (default): UI-driven tests — navigate pages, interact with elements (click, type, select), verify visible state. Never uses `evaluate` for API calls.
|
|
40
|
+
- **`api`**: Backend API tests — use `evaluate` actions for GraphQL/REST calls, assert response shapes and values. No UI interaction needed.
|
|
41
|
+
|
|
42
|
+
CLI: `e2e-runner issue <url> --generate --test-type api`
|
|
43
|
+
MCP: `e2e_issue({ url, testType: "api" })`
|
|
44
|
+
|
|
45
|
+
## GitLab Limitations
|
|
46
|
+
|
|
47
|
+
- Requires `glab` CLI installed and authenticated (`glab auth login`)
|
|
48
|
+
- Self-hosted GitLab instances supported via URL detection
|
|
49
|
+
- Verify mode works with GitLab issues but does NOT post comments back to the issue
|
|
50
|
+
- Private repos require `glab` to be authenticated with appropriate access
|
|
51
|
+
- The `authToken`/`authStorageKey` params on `e2e_issue` are for the **app under test**, not for GitLab API auth
|
|
52
|
+
|
|
53
|
+
## Key Files
|
|
54
|
+
|
|
55
|
+
| File | Purpose |
|
|
56
|
+
|------|---------|
|
|
57
|
+
| `src/issues.js` | GitHub/GitLab provider drivers |
|
|
58
|
+
| `src/ai-generate.js` | AI prompt builder + Claude API |
|
|
59
|
+
| `src/verify.js` | Verification orchestrator: fetch + generate + run |
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Multi-Pool Support
|
|
2
|
+
|
|
3
|
+
A single runner can distribute tests across multiple Chrome pools on different machines. `src/pool-manager.js` abstracts pool selection behind a **least-pressure** strategy. `poolUrl` (singular string) remains fully backwards compatible.
|
|
4
|
+
|
|
5
|
+
## Configuration
|
|
6
|
+
|
|
7
|
+
```js
|
|
8
|
+
// Single pool (unchanged)
|
|
9
|
+
export default { poolUrl: 'ws://localhost:3333' };
|
|
10
|
+
|
|
11
|
+
// Multi-pool — tests distribute across all pools
|
|
12
|
+
export default {
|
|
13
|
+
poolUrls: ['ws://machine1:3333', 'ws://machine2:3333'],
|
|
14
|
+
concurrency: 15,
|
|
15
|
+
};
|
|
16
|
+
|
|
17
|
+
// CI environment profile
|
|
18
|
+
export default {
|
|
19
|
+
poolUrl: 'ws://localhost:3333',
|
|
20
|
+
environments: {
|
|
21
|
+
ci: { poolUrls: ['ws://ci-chrome1:3333', 'ws://ci-chrome2:3333'], concurrency: 10 },
|
|
22
|
+
},
|
|
23
|
+
};
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
**CLI:** `e2e-runner run --all --pool-urls ws://m1:3333,ws://m2:3333`
|
|
27
|
+
**Env:** `CHROME_POOL_URLS=ws://m1:3333,ws://m2:3333` (comma-separated)
|
|
28
|
+
|
|
29
|
+
## Pool Selection Algorithm (`selectPool`)
|
|
30
|
+
|
|
31
|
+
1. Query all pools' `/pressure` in parallel
|
|
32
|
+
2. Filter to reachable pools with `running < maxConcurrent`
|
|
33
|
+
3. Sort by: lowest `running/maxConcurrent` → fewest queued → most free slots
|
|
34
|
+
4. Return best candidate URL
|
|
35
|
+
5. If all full, poll every 2s up to 60s, then pick least-pressured anyway
|
|
36
|
+
|
|
37
|
+
## Pool-Aware Queue
|
|
38
|
+
|
|
39
|
+
Before opening a browser connection, each worker checks the pool's `/pressure` endpoint. If the pool is at capacity, the worker waits (polling every 2s, up to 60s) for a free slot instead of piling requests into browserless's internal queue. This prevents memory pressure and SIGKILL of Chrome processes under heavy load.
|
|
40
|
+
|
|
41
|
+
## Failure Resilience
|
|
42
|
+
|
|
43
|
+
Dead pools are excluded from selection. `waitForAnyPool` succeeds if any pool responds. Failed connections trigger re-selection on the next worker.
|
|
44
|
+
|
|
45
|
+
## Config Normalization in `loadConfig()`
|
|
46
|
+
|
|
47
|
+
- If `poolUrls` array is set → `config._poolUrls = poolUrls`, `config.poolUrl = poolUrls[0]`
|
|
48
|
+
- Else → `config._poolUrls = [config.poolUrl]`
|
|
49
|
+
- `config.poolUrls` is deleted after normalization (use `config._poolUrls` internally)
|
|
50
|
+
|
|
51
|
+
## Key Functions (`src/pool-manager.js`)
|
|
52
|
+
|
|
53
|
+
| Function | Purpose |
|
|
54
|
+
|----------|---------|
|
|
55
|
+
| `getPoolUrls(config)` | Returns `config._poolUrls` (always an array) |
|
|
56
|
+
| `getAllPoolStatuses(poolUrls)` | Fetches `/pressure` from all pools in parallel |
|
|
57
|
+
| `getAggregatedPoolStatus(poolUrls)` | Combined view: `totalRunning`, `totalMaxConcurrent`, per-pool details |
|
|
58
|
+
| `waitForAnyPool(poolUrls)` | Blocks until at least one pool is reachable + available |
|
|
59
|
+
| `selectPool(poolUrls)` | Picks pool with lowest pressure ratio |
|
|
60
|
+
| `selectAndConnect(config)` | Convenience: selectPool + connectToPool in one call |
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Network Debugging Reference
|
|
2
|
+
|
|
3
|
+
## Network Error Handling
|
|
4
|
+
|
|
5
|
+
### `assert_no_network_errors` action
|
|
6
|
+
|
|
7
|
+
Checks accumulated `requestfailed` events during the test. If any network errors exist (e.g., `net::ERR_CONNECTION_REFUSED`), the test fails with details of each error URL. Place after critical page loads.
|
|
8
|
+
|
|
9
|
+
### `failOnNetworkError` config option
|
|
10
|
+
|
|
11
|
+
When `true`, automatically fails any test that has network errors after all actions complete. Default: `false` (opt-in).
|
|
12
|
+
|
|
13
|
+
Set via: config `failOnNetworkError: true` | CLI `--fail-on-network-error` | env `FAIL_ON_NETWORK_ERROR=true` | MCP `failOnNetworkError: true`
|
|
14
|
+
|
|
15
|
+
### `networkIgnoreDomains` config option
|
|
16
|
+
|
|
17
|
+
Array of domain substrings to filter from network error tracking. Errors from matching URLs are silently dropped by both `assert_no_network_errors` and `failOnNetworkError`.
|
|
18
|
+
|
|
19
|
+
Set via: config `networkIgnoreDomains: ['google-analytics.com', 'fonts.googleapis.com']` | CLI `--network-ignore-domains ga.com,fonts.com` | env `NETWORK_IGNORE_DOMAINS=ga.com,fonts.com` (comma-separated)
|
|
20
|
+
|
|
21
|
+
## Network Request/Response Logging
|
|
22
|
+
|
|
23
|
+
All XHR/fetch requests are captured with full detail regardless of status code:
|
|
24
|
+
|
|
25
|
+
- `url`, `method`, `status`, `statusText`, `duration`
|
|
26
|
+
- `requestHeaders` — all request headers as object
|
|
27
|
+
- `requestBody` — POST body (from `req.postData()`)
|
|
28
|
+
- `responseHeaders` — all response headers as object
|
|
29
|
+
- `responseBody` — full response text (truncated at 50KB)
|
|
30
|
+
|
|
31
|
+
Response bodies are read asynchronously and flushed via `Promise.allSettled` before the browser disconnects. Data is stored in the `network_logs` column in SQLite and displayed in the dashboard.
|
|
32
|
+
|
|
33
|
+
## MCP Response Optimization
|
|
34
|
+
|
|
35
|
+
The `e2e_run` MCP tool returns a compact `networkSummary` instead of full logs (~5KB vs ~400KB):
|
|
36
|
+
|
|
37
|
+
```json
|
|
38
|
+
{
|
|
39
|
+
"networkSummary": [{
|
|
40
|
+
"name": "test-name",
|
|
41
|
+
"totalRequests": 37,
|
|
42
|
+
"statusDistribution": { "2xx": 30, "3xx": 5, "4xx": 1, "5xx": 0, "other": 1 },
|
|
43
|
+
"avgDurationMs": 245,
|
|
44
|
+
"failedRequests": [{ "url": "/api/x", "method": "POST", "status": 500 }],
|
|
45
|
+
"slowestRequests": [{ "url": "/api/y", "method": "GET", "status": 200, "duration": 1200 }]
|
|
46
|
+
}]
|
|
47
|
+
}
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Drill-Down Pattern
|
|
51
|
+
|
|
52
|
+
The response includes `runDbId` — the SQLite row ID. Use it with `e2e_network_logs` to drill down:
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
1. e2e_run → compact summary + runDbId
|
|
56
|
+
2. e2e_network_logs(runDbId) → all requests (url, method, status, duration)
|
|
57
|
+
3. e2e_network_logs(runDbId, errorsOnly: true) → only failed requests
|
|
58
|
+
4. e2e_network_logs(runDbId, includeHeaders: true) → with headers
|
|
59
|
+
5. e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Dashboard REST equivalent: `GET /api/db/runs/:id/network-logs?testName=X&errorsOnly=true&includeHeaders=true`
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
# Test JSON Format Reference
|
|
2
|
+
|
|
3
|
+
## Basic Format (Array)
|
|
4
|
+
|
|
5
|
+
A test file is a JSON array of test objects:
|
|
6
|
+
|
|
7
|
+
```json
|
|
8
|
+
[
|
|
9
|
+
{
|
|
10
|
+
"name": "test-name",
|
|
11
|
+
"actions": [
|
|
12
|
+
{ "type": "goto", "value": "/page" },
|
|
13
|
+
{ "type": "assert_text", "text": "Expected content" }
|
|
14
|
+
]
|
|
15
|
+
}
|
|
16
|
+
]
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## Object Format (with Hooks)
|
|
20
|
+
|
|
21
|
+
When hooks are needed, use the object format:
|
|
22
|
+
|
|
23
|
+
```json
|
|
24
|
+
{
|
|
25
|
+
"hooks": {
|
|
26
|
+
"beforeAll": [{ "type": "goto", "value": "/setup" }],
|
|
27
|
+
"beforeEach": [{ "type": "goto", "value": "/" }],
|
|
28
|
+
"afterEach": [],
|
|
29
|
+
"afterAll": []
|
|
30
|
+
},
|
|
31
|
+
"tests": [
|
|
32
|
+
{ "name": "test-1", "actions": [...] }
|
|
33
|
+
]
|
|
34
|
+
}
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
**Hook lifecycle:**
|
|
38
|
+
- `beforeAll` — runs once before all tests (on a separate browser page, state does NOT carry over)
|
|
39
|
+
- `beforeEach` — runs before each individual test (on the test's own page)
|
|
40
|
+
- `afterEach` — runs after each test
|
|
41
|
+
- `afterAll` — runs once after all tests
|
|
42
|
+
|
|
43
|
+
> **Warning**: `beforeAll` runs on a separate page that closes before tests start. Don't use it for browser state setup (cookies, localStorage). Use `beforeEach` instead.
|
|
44
|
+
|
|
45
|
+
## Test Options
|
|
46
|
+
|
|
47
|
+
| Field | Type | Description |
|
|
48
|
+
|-------|------|-------------|
|
|
49
|
+
| `name` | string | **Required.** Test identifier. |
|
|
50
|
+
| `actions` | array | **Required.** Sequential browser actions. |
|
|
51
|
+
| `expect` | string | Visual verification description. Triggers auto-screenshot + AI judgment. |
|
|
52
|
+
| `serial` | boolean | Run sequentially after all parallel tests (for shared state). |
|
|
53
|
+
| `retries` | number | Per-test retry count on failure. Overrides global config. |
|
|
54
|
+
| `timeout` | number | Per-test timeout in ms. Overrides global `testTimeout` (default 60000). |
|
|
55
|
+
|
|
56
|
+
## Serial Tests
|
|
57
|
+
|
|
58
|
+
Tests that share mutable state should be marked serial to prevent race conditions:
|
|
59
|
+
|
|
60
|
+
```json
|
|
61
|
+
{ "name": "create-record", "serial": true, "actions": [...] },
|
|
62
|
+
{ "name": "verify-record", "serial": true, "actions": [...] }
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Serial tests run one-at-a-time **after** all parallel tests finish.
|
|
66
|
+
|
|
67
|
+
## Retry Behavior
|
|
68
|
+
|
|
69
|
+
### Test-level retries
|
|
70
|
+
```json
|
|
71
|
+
{ "name": "flaky-test", "retries": 3, "actions": [...] }
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Or globally: `--retries 2` / `retries: 2` in config. Each retry gets its own timeout. Flaky tests (pass after retry) are logged as "flaky".
|
|
75
|
+
|
|
76
|
+
### Action-level retries
|
|
77
|
+
```json
|
|
78
|
+
{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Or globally: `--action-retries 2`. Delay between action retries: `actionRetryDelay` (default 500ms).
|
|
82
|
+
|
|
83
|
+
## Reusable Modules
|
|
84
|
+
|
|
85
|
+
Create modules with `e2e_create_module`, reference them in tests:
|
|
86
|
+
|
|
87
|
+
```json
|
|
88
|
+
{
|
|
89
|
+
"name": "login-test",
|
|
90
|
+
"actions": [
|
|
91
|
+
{ "$use": "auth-login", "params": { "email": "admin@test.com", "password": "secret" } },
|
|
92
|
+
{ "type": "assert_url", "value": "/dashboard" }
|
|
93
|
+
]
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Module definition (in `e2e/modules/auth-login.json`):
|
|
98
|
+
```json
|
|
99
|
+
{
|
|
100
|
+
"$module": "auth-login",
|
|
101
|
+
"description": "Log in with email/password",
|
|
102
|
+
"params": {
|
|
103
|
+
"email": { "required": true, "description": "User email" },
|
|
104
|
+
"password": { "required": true, "description": "User password" }
|
|
105
|
+
},
|
|
106
|
+
"actions": [
|
|
107
|
+
{ "type": "goto", "value": "/login" },
|
|
108
|
+
{ "type": "type", "selector": "#email", "value": "{{email}}" },
|
|
109
|
+
{ "type": "type", "selector": "#password", "value": "{{password}}" },
|
|
110
|
+
{ "type": "click", "text": "Sign In" },
|
|
111
|
+
{ "type": "wait", "selector": ".dashboard" }
|
|
112
|
+
]
|
|
113
|
+
}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## Suite Naming & Ordering
|
|
117
|
+
|
|
118
|
+
Files can have numeric prefixes for execution order:
|
|
119
|
+
- `01-auth.json`, `02-dashboard.json`, `03-settings.json`
|
|
120
|
+
|
|
121
|
+
The `--suite` flag strips the prefix when matching: `--suite auth` finds `01-auth.json`.
|
|
122
|
+
|
|
123
|
+
## Excluding Tests
|
|
124
|
+
|
|
125
|
+
Use `exclude` in config to skip files when running `--all`:
|
|
126
|
+
|
|
127
|
+
```js
|
|
128
|
+
// e2e.config.js
|
|
129
|
+
export default {
|
|
130
|
+
exclude: ['explore-*', 'debug-*', 'draft-*']
|
|
131
|
+
};
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Individual `--suite` runs are not affected by exclude patterns.
|
|
135
|
+
|
|
136
|
+
## Environment Profiles
|
|
137
|
+
|
|
138
|
+
Define named profiles in config:
|
|
139
|
+
|
|
140
|
+
```js
|
|
141
|
+
// e2e.config.js
|
|
142
|
+
export default {
|
|
143
|
+
baseUrl: 'http://host.docker.internal:3000',
|
|
144
|
+
environments: {
|
|
145
|
+
staging: { baseUrl: 'https://staging.example.com' },
|
|
146
|
+
production: { baseUrl: 'https://example.com', concurrency: 5 }
|
|
147
|
+
}
|
|
148
|
+
};
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
Activate with `--env staging` or `E2E_ENV=staging`. Profile values override all other config.
|
|
152
|
+
|
|
153
|
+
## CI Output Formats
|
|
154
|
+
|
|
155
|
+
Use `--output <format>` to control report output: `json` (default), `junit` (JUnit XML), or `both`. JUnit XML is generated without external dependencies and saved to `{screenshotsDir}/junit.xml`. The `generateJUnitXML()` function is also exported from the programmatic API.
|
|
156
|
+
|
|
157
|
+
## Config Priority (ascending)
|
|
158
|
+
|
|
159
|
+
1. Hardcoded defaults
|
|
160
|
+
2. `e2e.config.js` or `e2e.config.json`
|
|
161
|
+
3. Environment variables (`BASE_URL`, `CONCURRENCY`, etc.)
|
|
162
|
+
4. CLI flags (`--base-url`, `--concurrency`, etc.)
|
|
163
|
+
5. Environment profile merge (via `--env`)
|
|
@@ -0,0 +1,224 @@
|
|
|
1
|
+
# Troubleshooting Guide
|
|
2
|
+
|
|
3
|
+
## Pool Connection Issues
|
|
4
|
+
|
|
5
|
+
### "Pool not reachable" / Connection refused
|
|
6
|
+
|
|
7
|
+
**Cause**: Chrome pool (browserless/chrome Docker container) is not running.
|
|
8
|
+
|
|
9
|
+
**Fix**:
|
|
10
|
+
```bash
|
|
11
|
+
npx e2e-runner pool start
|
|
12
|
+
npx e2e-runner pool status # verify it's running
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Pool management is CLI-only — `pool start` and `pool stop` are not available via MCP.
|
|
16
|
+
|
|
17
|
+
### "Pool at capacity" / Tests queuing
|
|
18
|
+
|
|
19
|
+
**Cause**: All Chrome sessions are occupied.
|
|
20
|
+
|
|
21
|
+
**Fix**: Increase capacity or reduce concurrency:
|
|
22
|
+
```bash
|
|
23
|
+
npx e2e-runner pool stop
|
|
24
|
+
npx e2e-runner pool start --max-sessions 10
|
|
25
|
+
```
|
|
26
|
+
Or reduce test concurrency: `--concurrency 2`
|
|
27
|
+
|
|
28
|
+
The runner checks `/pressure` before each connection and waits up to 60s for a free slot.
|
|
29
|
+
|
|
30
|
+
### Docker not running
|
|
31
|
+
|
|
32
|
+
**Cause**: Docker daemon is not started.
|
|
33
|
+
|
|
34
|
+
**Fix**: Start Docker Desktop or `sudo systemctl start docker`, then `npx e2e-runner pool start`.
|
|
35
|
+
|
|
36
|
+
## React / SPA Issues
|
|
37
|
+
|
|
38
|
+
### React inputs not updating state
|
|
39
|
+
|
|
40
|
+
**Symptom**: `type` action enters text but React state doesn't change (form validation fails, submit disabled).
|
|
41
|
+
|
|
42
|
+
**Fix**: Use `type_react` instead of `type` for React controlled inputs:
|
|
43
|
+
```json
|
|
44
|
+
{ "type": "type_react", "selector": "#email", "value": "user@test.com" }
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
`type_react` uses the native value setter and dispatches `input` + `change` events that React's synthetic event system recognizes.
|
|
48
|
+
|
|
49
|
+
### SPA navigation not completing
|
|
50
|
+
|
|
51
|
+
**Symptom**: `goto` hangs or times out on client-side route changes.
|
|
52
|
+
|
|
53
|
+
**Fix**: Use `navigate` instead of `goto` for SPA route changes:
|
|
54
|
+
```json
|
|
55
|
+
{ "type": "navigate", "value": "/new-page" }
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
`navigate` uses a 5s race timeout and won't block if `load` doesn't fire (common in SPAs).
|
|
59
|
+
|
|
60
|
+
### MUI autocomplete not opening
|
|
61
|
+
|
|
62
|
+
**Symptom**: Clicking or typing in an MUI Autocomplete doesn't open the dropdown.
|
|
63
|
+
|
|
64
|
+
**Fix**: Use `focus_autocomplete` to properly focus by label text:
|
|
65
|
+
```json
|
|
66
|
+
{ "type": "focus_autocomplete", "text": "Search by name" },
|
|
67
|
+
{ "type": "type_react", "selector": "#autocomplete-input", "value": "search term" },
|
|
68
|
+
{ "type": "click_option", "text": "Desired option" }
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Flaky Tests
|
|
72
|
+
|
|
73
|
+
### Intermittent failures on dynamic content
|
|
74
|
+
|
|
75
|
+
**Symptom**: Tests pass sometimes, fail others. Usually timing-related.
|
|
76
|
+
|
|
77
|
+
**Fixes**:
|
|
78
|
+
1. Add explicit `wait` before assertions:
|
|
79
|
+
```json
|
|
80
|
+
{ "type": "wait", "selector": ".data-loaded" },
|
|
81
|
+
{ "type": "assert_text", "text": "Expected content" }
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
2. Use action-level retries for known flaky selectors:
|
|
85
|
+
```json
|
|
86
|
+
{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
3. Use test-level retries:
|
|
90
|
+
```json
|
|
91
|
+
{ "name": "flaky-test", "retries": 2, "actions": [...] }
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
4. Check the learning system for patterns:
|
|
95
|
+
```
|
|
96
|
+
e2e_learnings("flaky") → identify consistently flaky tests
|
|
97
|
+
e2e_learnings("selectors") → find unstable selectors
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
### Tests interfering with each other
|
|
101
|
+
|
|
102
|
+
**Symptom**: Tests pass individually but fail when run together.
|
|
103
|
+
|
|
104
|
+
**Fix**: Mark tests that share mutable state as `serial`:
|
|
105
|
+
```json
|
|
106
|
+
{ "name": "create-item", "serial": true, "actions": [...] },
|
|
107
|
+
{ "name": "verify-item", "serial": true, "actions": [...] }
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
## Timeout Issues
|
|
111
|
+
|
|
112
|
+
### Test timeout (default 60s)
|
|
113
|
+
|
|
114
|
+
**Fix**: Increase per-test or globally:
|
|
115
|
+
```json
|
|
116
|
+
{ "name": "slow-test", "timeout": 120000, "actions": [...] }
|
|
117
|
+
```
|
|
118
|
+
Or globally: `--test-timeout 120000`
|
|
119
|
+
|
|
120
|
+
### Action timeout (default 10s)
|
|
121
|
+
|
|
122
|
+
Each action's `waitForSelector` uses the default timeout. Override per-action:
|
|
123
|
+
```json
|
|
124
|
+
{ "type": "wait", "selector": ".slow-element", "timeout": 30000 }
|
|
125
|
+
```
|
|
126
|
+
Or globally: `--timeout 30000`
|
|
127
|
+
|
|
128
|
+
## Network Errors
|
|
129
|
+
|
|
130
|
+
### Tests passing but network requests failing
|
|
131
|
+
|
|
132
|
+
**Symptom**: Tests pass but `networkSummary` shows failed requests.
|
|
133
|
+
|
|
134
|
+
**Fix**: Enable strict mode to fail tests with network errors:
|
|
135
|
+
```
|
|
136
|
+
e2e_run({ all: true, failOnNetworkError: true })
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Or use `assert_no_network_errors` at specific points:
|
|
140
|
+
```json
|
|
141
|
+
{ "type": "goto", "value": "/api-heavy-page" },
|
|
142
|
+
{ "type": "wait", "selector": ".loaded" },
|
|
143
|
+
{ "type": "assert_no_network_errors" }
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
### Investigating specific failures
|
|
147
|
+
|
|
148
|
+
Use network log drill-down:
|
|
149
|
+
```
|
|
150
|
+
e2e_network_logs(runDbId, errorsOnly: true) → see all failed requests
|
|
151
|
+
e2e_network_logs(runDbId, urlPattern: "/api/users") → filter by URL
|
|
152
|
+
e2e_network_logs(runDbId, testName: "create-user", includeBodies: true) → full request/response
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
## Common Mistakes
|
|
156
|
+
|
|
157
|
+
### Using `beforeAll` for browser state
|
|
158
|
+
|
|
159
|
+
`beforeAll` runs on a separate page that closes before tests. Use `beforeEach` for state setup.
|
|
160
|
+
|
|
161
|
+
### Using `evaluate` for simple assertions
|
|
162
|
+
|
|
163
|
+
Prefer granular assertion actions over `evaluate` with inline JS:
|
|
164
|
+
```json
|
|
165
|
+
// Bad: verbose, error-prone
|
|
166
|
+
{ "type": "evaluate", "value": "if (!document.querySelector('h1').textContent.includes('Dashboard')) throw 'not found'" }
|
|
167
|
+
|
|
168
|
+
// Good: clear, auto-waits
|
|
169
|
+
{ "type": "assert_element_text", "selector": "h1", "text": "Dashboard" }
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
### Forgetting `cwd` in MCP calls
|
|
173
|
+
|
|
174
|
+
All MCP tools need `cwd` to resolve config files and test directories. Always pass the project root.
|
|
175
|
+
|
|
176
|
+
### Path-only `assert_url`
|
|
177
|
+
|
|
178
|
+
When checking paths, use path-only format (starts with `/`):
|
|
179
|
+
```json
|
|
180
|
+
{ "type": "assert_url", "value": "/dashboard" }
|
|
181
|
+
```
|
|
182
|
+
This compares against the pathname only, ignoring the `host.docker.internal` origin.
|
|
183
|
+
|
|
184
|
+
## Action Type Pre-Validation
|
|
185
|
+
|
|
186
|
+
All action types are validated at **load time** (before any browser connections). If a test file contains an unknown action type (e.g., a typo like `"clik"`), loading throws immediately with the location:
|
|
187
|
+
|
|
188
|
+
```
|
|
189
|
+
Unknown action type(s) in auth.json: "clik" in test "login-test"
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
The `KNOWN_ACTION_TYPES` Set in `src/actions.js` is the single source of truth. Unknown actions also throw at runtime as a safety net.
|
|
193
|
+
|
|
194
|
+
## Screenshot Hashes
|
|
195
|
+
|
|
196
|
+
Every screenshot captured during a run is assigned a short hash (`ss:a3f2b1c9`) — the first 8 hex chars of the SHA-256 of its file path. Hashes are deterministic and computed identically on the server (Node `crypto`) and in the browser (Web Crypto API).
|
|
197
|
+
|
|
198
|
+
**Flow**: screenshot saved on disk → `saveRun()` registers hash in SQLite `screenshot_hashes` table → dashboard shows `[ss:XXXXXXXX]` badge (click to copy) → user pastes hash in Claude Code → `e2e_screenshot` MCP tool looks up hash, reads file, returns the image.
|
|
199
|
+
|
|
200
|
+
- Hashes are registered inside the `saveRun()` transaction (covers action, error, verification, and baseline screenshots)
|
|
201
|
+
- The `ss:` prefix is optional when calling `e2e_screenshot` — stripped during lookup
|
|
202
|
+
- Dashboard computes hashes client-side (Web Crypto) for the Live view (before `persistRun()` writes to DB)
|
|
203
|
+
- Run detail API (`/api/db/runs/:id`) includes `screenshotHashes` map per test result
|
|
204
|
+
- Dashboard endpoint `/api/screenshot-hash/:hash` serves the image by hash
|
|
205
|
+
- Dashboard Screenshots view has a **search bar** — type a hash to find and display the screenshot
|
|
206
|
+
|
|
207
|
+
## Web Dashboard
|
|
208
|
+
|
|
209
|
+
**`src/dashboard.js`** — HTTP server, REST API, WebSocket broadcast, pool polling.
|
|
210
|
+
**`templates/dashboard.html`** — SPA, dark theme, vanilla JS, safe DOM (textContent + createEl helper).
|
|
211
|
+
|
|
212
|
+
**Features:**
|
|
213
|
+
- Live test execution with WebSocket updates
|
|
214
|
+
- Run history with inline detail expansion
|
|
215
|
+
- Screenshots gallery with hash badges and hash search
|
|
216
|
+
- Network request logs with clickable expandable rows (full request/response detail)
|
|
217
|
+
- Pool status monitoring
|
|
218
|
+
- Multi-project support via project selector
|
|
219
|
+
- Variables tab with masked values, inline edit, add, and delete
|
|
220
|
+
|
|
221
|
+
**CLI:** `e2e-runner dashboard [--port 8484]`
|
|
222
|
+
**MCP tools:** `e2e_dashboard_start`, `e2e_dashboard_stop`
|
|
223
|
+
|
|
224
|
+
Config defaults: `dashboardPort: 8484`, `maxHistoryRuns: 100`
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Variables Reference
|
|
2
|
+
|
|
3
|
+
Variables replace hardcoded sensitive values (JWT tokens, user IDs, API keys, etc.) in test JSON. Stored in SQLite (`~/.e2e-runner/dashboard.db`), scoped per project and per suite, editable from the dashboard UI.
|
|
4
|
+
|
|
5
|
+
## Syntax
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
{{var.TOKEN}} → resolves from DB (suite scope → project scope)
|
|
9
|
+
{{env.MY_VAR}} → resolves from process.env
|
|
10
|
+
{{param}} → existing module param substitution (unchanged)
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
**Resolution priority:** suite vars > project vars > error if not found.
|
|
14
|
+
|
|
15
|
+
## Usage in Test JSON
|
|
16
|
+
|
|
17
|
+
```json
|
|
18
|
+
{ "$use": "auth-jwt", "params": { "token": "{{var.JWT_TOKEN}}", "orgId": "{{var.ORG_ID}}" } }
|
|
19
|
+
{ "type": "goto", "value": "/users/{{var.USER_ID}}/profile" }
|
|
20
|
+
{ "type": "gql", "value": "{ user(id: \"{{var.USER_ID}}\") { name } }" }
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## MCP Tool (`e2e_vars`)
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
e2e_vars({ action: "set", key: "TOKEN", value: "abc123", scope: "project" })
|
|
27
|
+
e2e_vars({ action: "set", key: "TOKEN", value: "xyz789", scope: "auth" }) // suite-specific override
|
|
28
|
+
e2e_vars({ action: "list" })
|
|
29
|
+
e2e_vars({ action: "get", key: "TOKEN" })
|
|
30
|
+
e2e_vars({ action: "delete", key: "TOKEN", scope: "project" })
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Dashboard UI
|
|
34
|
+
|
|
35
|
+
Variables tab shows all variables grouped by scope. Values are masked by default (click to reveal). Inline edit, add new, and delete are supported.
|
|
36
|
+
|
|
37
|
+
## REST API
|
|
38
|
+
|
|
39
|
+
- `GET /api/db/projects/:id/variables` — list all vars for project
|
|
40
|
+
- `PUT /api/db/projects/:id/variables` — set a variable `{ scope, key, value }`
|
|
41
|
+
- `DELETE /api/db/projects/:id/variables/:scope/:key` — delete a variable
|