@matware/e2e-runner 1.1.1 → 1.3.0

package/skills/e2e-testing/references/auth-strategies.md

@@ -0,0 +1,91 @@
+# Authentication Strategies
+
+Tests can authenticate using multiple strategies depending on the app's auth mechanism.
+
+## 1. UI Login (universal)
+
+Fill the login form in `beforeEach`. Works with any auth system — the browser stores cookies/tokens automatically.
+
+```json
+{ "hooks": { "beforeEach": [
+  { "type": "goto", "value": "/login" },
+  { "type": "type", "selector": "#email", "value": "test@example.com" },
+  { "type": "type", "selector": "#password", "value": "secret" },
+  { "type": "click", "text": "Sign In" },
+  { "type": "wait", "selector": ".dashboard" }
+]}}
+```
+
+## 2. JWT Injection (SPAs)
+
+Skip the login form by injecting a token into `localStorage` or `sessionStorage`:
+
+```json
+{ "type": "set_storage", "value": "accessToken=eyJhbG..." }
+{ "type": "set_storage", "value": "token=eyJhbG...", "selector": "session" }
+```
+
+## 3. Config-Level Token
+
+Set `authToken` + `authStorageKey` in config, env vars (`AUTH_TOKEN`, `AUTH_STORAGE_KEY`), or CLI flags (`--auth-token`, `--auth-storage-key`). Used by `e2e_capture` and `e2e_issue --verify`.
+
+## 4. Cookie-Based (server-rendered)
+
+Set cookies via `evaluate`:
+```json
+{ "type": "evaluate", "value": "document.cookie = 'session_id=abc123; path=/; SameSite=Lax'" }
+```
+
+## 5. API Headers
+
+Override `fetch` to inject Authorization headers (useful for `--test-type api`):
+```json
+{ "type": "evaluate", "value": "const orig = window.fetch; window.fetch = (url, opts = {}) => { opts.headers = { ...opts.headers, 'Authorization': 'Bearer eyJhbG...' }; return orig(url, opts); }" }
+```
+
+## 6. OAuth/SSO
+
+Use a test-environment bypass endpoint, pre-authenticated token injection, or CI-obtained session cookie. External OAuth providers (Google, GitHub, Okta) can't be automated directly.
+
+## Auth Auto-Login
+
+Automatically fetch an auth token from a login API endpoint before tests run. Avoids repeating login flows in every test.
+
+### Config fields
+
+| Field | Description | Env / CLI |
+|-------|-------------|-----------|
+| `authLoginEndpoint` | Full URL to POST credentials to | `AUTH_LOGIN_ENDPOINT` / `--auth-login-endpoint` |
+| `authCredentials` | Object with login credentials (config file only — never in env vars) | — |
+| `authTokenPath` | Dot-path to extract token from response JSON (default: `'token'`) | `AUTH_TOKEN_PATH` / `--auth-token-path` |
+
+### Flow
+
+1. Before workers start, if `authLoginEndpoint` is set and `authToken` is NOT already set, `fetchAuthToken()` POSTs `authCredentials` as JSON.
+2. The token is extracted from the response using `authTokenPath` (supports dot notation, e.g. `'data.access_token'`).
+3. The extracted token is stored in `config.authToken`.
+4. Each test worker navigates to `baseUrl` and injects `localStorage[authStorageKey] = authToken` before `beforeEach` hooks run.
+
+### Example config
+
+```js
+export default {
+  authLoginEndpoint: 'https://api.example.com/auth/login',
+  authCredentials: { email: 'test@example.com', password: 'secret123' },
+  authTokenPath: 'data.access_token',
+  authStorageKey: 'accessToken',
+};
+```
+
+If `authToken` is already set (via config, env var, or CLI), the auto-login step is skipped.
+
+## Clearing State
+
+Each test runs in a fresh browser context (auto-clean). Use `clear_cookies` to explicitly clear cookies + localStorage + sessionStorage mid-test.
+
+## Reusable Auth Modules
+
+Create `e2e/modules/login.json` or `e2e/modules/auth-token.json` and reference via:
+```json
+{ "$use": "login", "params": { "email": "...", "password": "..." } }
+```

package/skills/e2e-testing/references/graphql.md

@@ -0,0 +1,59 @@
+# GraphQL Action Reference
+
+The `gql` action executes GraphQL queries and mutations via browser `fetch`, with automatic auth token injection from `localStorage`. It also installs a `window.__e2eGql(query, vars)` helper for use in subsequent `evaluate` actions.
+
+## Action Fields
+
+| Action | Fields | Behavior |
+|--------|--------|----------|
+| `gql` | `value` (query string, required), `text` (variables JSON, optional), `selector` (assertion JS expression, optional) | Sends a GraphQL request to the configured endpoint. Reads auth token from localStorage. Throws on GraphQL errors. Returns `{ value: response.data }`. Stores full response on `window.__e2eLastGql`. |
+
+## Config Fields
+
+| Field | Default | Env Var | CLI Flag |
+|-------|---------|---------|----------|
+| `gqlEndpoint` | `/api/graphql` | `GQL_ENDPOINT` | `--gql-endpoint` |
+| `gqlAuthHeader` | `Authorization` | `GQL_AUTH_HEADER` | `--gql-auth-header` |
+| `gqlAuthKey` | `accessToken` | `GQL_AUTH_KEY` | `--gql-auth-key` |
+| `gqlAuthPrefix` | `Bearer ` | `GQL_AUTH_PREFIX` | `--gql-auth-prefix` |
+
+The endpoint path is appended to `location.origin`. The auth token is read from `localStorage[gqlAuthKey]` and sent as `gqlAuthHeader: gqlAuthPrefix + token`.
+
+## Examples
+
+### Basic query
+```json
+{ "type": "gql", "value": "{ users { id name } }" }
+```
+
+### With variables
+```json
+{ "type": "gql", "value": "query($id: ID, $orgId: ID) { orders(userId: $id, orgId: $orgId) { orderId status } }", "text": "{\"id\": \"abc-123\", \"orgId\": \"org-456\"}" }
+```
+
+### With inline assertion (selector field)
+```json
+// selector is a JS expression where `r` is the full GraphQL response
+{ "type": "gql", "value": "{ pendingOrders(userId: \"abc-123\") { status } }", "selector": "r.data.pendingOrders.some(o => o.status === 'CANCELLED') ? 'FAIL: cancelled order found in pending list' : 'OK: all pending'" }
+```
+
+### Using the installed helper in evaluate
+```json
+// After any gql action runs, window.__e2eGql(query, vars) is available
+{ "type": "gql", "value": "{ __typename }" }
+{ "type": "evaluate", "value": "(async () => { const r = await window.__e2eGql('query { orders(status: [PROCESSING]) { orderId } }'); for (const o of r.data.orders) await window.__e2eGql('mutation($id: ID, $input: OrderInput) { updateOrder(orderId: $id, input: $input) { orderId } }', { id: o.orderId, input: { status: 'COMPLETED' } }); return 'Updated ' + r.data.orders.length; })()" }
+```
+
+## Custom Auth Header Config
+
+If your API uses a non-standard auth header (e.g., `x-api-key` instead of `Authorization`):
+
+```js
+// e2e.config.js
+export default {
+  gqlEndpoint: '/api/graphql',
+  gqlAuthHeader: 'x-api-key',   // custom header name
+  gqlAuthKey: 'apiToken',       // localStorage key to read from
+  gqlAuthPrefix: '',             // no 'Bearer ' prefix — raw token
+};
+```

package/skills/e2e-testing/references/issue-verification.md

@@ -0,0 +1,59 @@
+# Issue-to-Test Verification Reference
+
+Turns bug reports and feature requests into executable E2E tests.
+
+## Supported Providers
+
+- **GitHub** (`github.com`) — requires `gh` CLI (`gh auth login`)
+- **GitLab** (including self-hosted) — requires `glab` CLI (`glab auth login`)
+
+Auto-detected from URL. All external commands use `execFileSync` (no shell injection).
+
+## Two AI Modes
+
+### 1. Prompt Mode (default, no API key)
+
+`e2e_issue` MCP tool returns issue details + a structured prompt. Claude Code then uses `e2e_create_test` to create tests and `e2e_run` to execute them.
+
+### 2. Verify Mode (requires `ANTHROPIC_API_KEY`)
+
+Calls Claude API directly to generate tests, runs them, and reports whether the bug is confirmed or not reproducible.
+
+## Config Fields
+
+| Field | Description | Env Var |
+|-------|-------------|---------|
+| `anthropicApiKey` | Required for verify/generate mode | `ANTHROPIC_API_KEY` |
+| `anthropicModel` | Claude model for generation (default: `claude-sonnet-4-5-20250929`) | `ANTHROPIC_MODEL` |
+
+## Bug Verification Logic
+
+Generated tests assert **correct** behavior:
+- **Test failure** = bug confirmed (expected behavior doesn't match reality)
+- **All tests pass** = not reproducible
+
+## Test Categories (`testType`)
+
+The `--test-type` CLI flag (or `testType` MCP parameter) controls what kind of tests the AI generates:
+
+- **`e2e`** (default): UI-driven tests — navigate pages, interact with elements (click, type, select), verify visible state. Never uses `evaluate` for API calls.
+- **`api`**: Backend API tests — use `evaluate` actions for GraphQL/REST calls, assert response shapes and values. No UI interaction needed.
+
+CLI: `e2e-runner issue <url> --generate --test-type api`
+MCP: `e2e_issue({ url, testType: "api" })`
+
+## GitLab Limitations
+
+- Requires `glab` CLI installed and authenticated (`glab auth login`)
+- Self-hosted GitLab instances supported via URL detection
+- Verify mode works with GitLab issues but does NOT post comments back to the issue
+- Private repos require `glab` to be authenticated with appropriate access
+- The `authToken`/`authStorageKey` params on `e2e_issue` are for the **app under test**, not for GitLab API auth
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `src/issues.js` | GitHub/GitLab provider drivers |
+| `src/ai-generate.js` | AI prompt builder + Claude API |
+| `src/verify.js` | Verification orchestrator: fetch + generate + run |

package/skills/e2e-testing/references/multi-pool.md

@@ -0,0 +1,60 @@
+# Multi-Pool Support
+
+A single runner can distribute tests across multiple Chrome pools on different machines. `src/pool-manager.js` abstracts pool selection behind a **least-pressure** strategy. `poolUrl` (singular string) remains fully backwards compatible.
+
+## Configuration
+
+```js
+// Single pool (unchanged)
+export default { poolUrl: 'ws://localhost:3333' };
+
+// Multi-pool — tests distribute across all pools
+export default {
+  poolUrls: ['ws://machine1:3333', 'ws://machine2:3333'],
+  concurrency: 15,
+};
+
+// CI environment profile
+export default {
+  poolUrl: 'ws://localhost:3333',
+  environments: {
+    ci: { poolUrls: ['ws://ci-chrome1:3333', 'ws://ci-chrome2:3333'], concurrency: 10 },
+  },
+};
+```
+
+**CLI:** `e2e-runner run --all --pool-urls ws://m1:3333,ws://m2:3333`
+**Env:** `CHROME_POOL_URLS=ws://m1:3333,ws://m2:3333` (comma-separated)
+
+## Pool Selection Algorithm (`selectPool`)
+
+1. Query all pools' `/pressure` in parallel
+2. Filter to reachable pools with `running < maxConcurrent`
+3. Sort by: lowest `running/maxConcurrent` → fewest queued → most free slots
+4. Return best candidate URL
+5. If all full, poll every 2s up to 60s, then pick least-pressured anyway
+
+## Pool-Aware Queue
+
+Before opening a browser connection, each worker checks the pool's `/pressure` endpoint. If the pool is at capacity, the worker waits (polling every 2s, up to 60s) for a free slot instead of piling requests into browserless's internal queue. This prevents memory pressure and SIGKILL of Chrome processes under heavy load.
+
+## Failure Resilience
+
+Dead pools are excluded from selection. `waitForAnyPool` succeeds if any pool responds. Failed connections trigger re-selection on the next worker.
+
+## Config Normalization in `loadConfig()`
+
+- If `poolUrls` array is set → `config._poolUrls = poolUrls`, `config.poolUrl = poolUrls[0]`
+- Else → `config._poolUrls = [config.poolUrl]`
+- `config.poolUrls` is deleted after normalization (use `config._poolUrls` internally)
+
+## Key Functions (`src/pool-manager.js`)
+
+| Function | Purpose |
+|----------|---------|
+| `getPoolUrls(config)` | Returns `config._poolUrls` (always an array) |
+| `getAllPoolStatuses(poolUrls)` | Fetches `/pressure` from all pools in parallel |
+| `getAggregatedPoolStatus(poolUrls)` | Combined view: `totalRunning`, `totalMaxConcurrent`, per-pool details |
+| `waitForAnyPool(poolUrls)` | Blocks until at least one pool is reachable + available |
+| `selectPool(poolUrls)` | Picks pool with lowest pressure ratio |
+| `selectAndConnect(config)` | Convenience: selectPool + connectToPool in one call |

package/skills/e2e-testing/references/network-debugging.md

@@ -0,0 +1,62 @@
+# Network Debugging Reference
+
+## Network Error Handling
+
+### `assert_no_network_errors` action
+
+Checks accumulated `requestfailed` events during the test. If any network errors exist (e.g., `net::ERR_CONNECTION_REFUSED`), the test fails with details of each error URL. Place after critical page loads.
+
+### `failOnNetworkError` config option
+
+When `true`, automatically fails any test that has network errors after all actions complete. Default: `false` (opt-in).
+
+Set via: config `failOnNetworkError: true` | CLI `--fail-on-network-error` | env `FAIL_ON_NETWORK_ERROR=true` | MCP `failOnNetworkError: true`
+
+### `networkIgnoreDomains` config option
+
+Array of domain substrings to filter from network error tracking. Errors from matching URLs are silently dropped by both `assert_no_network_errors` and `failOnNetworkError`.
+
+Set via: config `networkIgnoreDomains: ['google-analytics.com', 'fonts.googleapis.com']` | CLI `--network-ignore-domains ga.com,fonts.com` | env `NETWORK_IGNORE_DOMAINS=ga.com,fonts.com` (comma-separated)
+
+## Network Request/Response Logging
+
+All XHR/fetch requests are captured with full detail regardless of status code:
+
+- `url`, `method`, `status`, `statusText`, `duration`
+- `requestHeaders` — all request headers as object
+- `requestBody` — POST body (from `req.postData()`)
+- `responseHeaders` — all response headers as object
+- `responseBody` — full response text (truncated at 50KB)
+
+Response bodies are read asynchronously and flushed via `Promise.allSettled` before the browser disconnects. Data is stored in the `network_logs` column in SQLite and displayed in the dashboard.
+
+## MCP Response Optimization
+
+The `e2e_run` MCP tool returns a compact `networkSummary` instead of full logs (~5KB vs ~400KB):
+
+```json
+{
+  "networkSummary": [{
+    "name": "test-name",
+    "totalRequests": 37,
+    "statusDistribution": { "2xx": 30, "3xx": 5, "4xx": 1, "5xx": 0, "other": 1 },
+    "avgDurationMs": 245,
+    "failedRequests": [{ "url": "/api/x", "method": "POST", "status": 500 }],
+    "slowestRequests": [{ "url": "/api/y", "method": "GET", "status": 200, "duration": 1200 }]
+  }]
+}
+```
+
+## Drill-Down Pattern
+
+The response includes `runDbId` — the SQLite row ID. Use it with `e2e_network_logs` to drill down:
+
+```
+1. e2e_run → compact summary + runDbId
+2. e2e_network_logs(runDbId) → all requests (url, method, status, duration)
+3. e2e_network_logs(runDbId, errorsOnly: true) → only failed requests
+4. e2e_network_logs(runDbId, includeHeaders: true) → with headers
+5. e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
+```
+
+Dashboard REST equivalent: `GET /api/db/runs/:id/network-logs?testName=X&errorsOnly=true&includeHeaders=true`

package/skills/e2e-testing/references/test-json-format.md

@@ -0,0 +1,163 @@
+# Test JSON Format Reference
+
+## Basic Format (Array)
+
+A test file is a JSON array of test objects:
+
+```json
+[
+  {
+    "name": "test-name",
+    "actions": [
+      { "type": "goto", "value": "/page" },
+      { "type": "assert_text", "text": "Expected content" }
+    ]
+  }
+]
+```
+
+## Object Format (with Hooks)
+
+When hooks are needed, use the object format:
+
+```json
+{
+  "hooks": {
+    "beforeAll": [{ "type": "goto", "value": "/setup" }],
+    "beforeEach": [{ "type": "goto", "value": "/" }],
+    "afterEach": [],
+    "afterAll": []
+  },
+  "tests": [
+    { "name": "test-1", "actions": [...] }
+  ]
+}
+```
+
+**Hook lifecycle:**
+- `beforeAll` — runs once before all tests (on a separate browser page, state does NOT carry over)
+- `beforeEach` — runs before each individual test (on the test's own page)
+- `afterEach` — runs after each test
+- `afterAll` — runs once after all tests
+
+> **Warning**: `beforeAll` runs on a separate page that closes before tests start. Don't use it for browser state setup (cookies, localStorage). Use `beforeEach` instead.
+
+## Test Options
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | string | **Required.** Test identifier. |
+| `actions` | array | **Required.** Sequential browser actions. |
+| `expect` | string | Visual verification description. Triggers auto-screenshot + AI judgment. |
+| `serial` | boolean | Run sequentially after all parallel tests (for shared state). |
+| `retries` | number | Per-test retry count on failure. Overrides global config. |
+| `timeout` | number | Per-test timeout in ms. Overrides global `testTimeout` (default 60000). |
+
+## Serial Tests
+
+Tests that share mutable state should be marked serial to prevent race conditions:
+
+```json
+{ "name": "create-record", "serial": true, "actions": [...] },
+{ "name": "verify-record", "serial": true, "actions": [...] }
+```
+
+Serial tests run one-at-a-time **after** all parallel tests finish.
+
+## Retry Behavior
+
+### Test-level retries
+```json
+{ "name": "flaky-test", "retries": 3, "actions": [...] }
+```
+
+Or globally: `--retries 2` / `retries: 2` in config. Each retry gets its own timeout. Flaky tests (pass after retry) are logged as "flaky".
+
+### Action-level retries
+```json
+{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
+```
+
+Or globally: `--action-retries 2`. Delay between action retries: `actionRetryDelay` (default 500ms).
+
+## Reusable Modules
+
+Create modules with `e2e_create_module`, reference them in tests:
+
+```json
+{
+  "name": "login-test",
+  "actions": [
+    { "$use": "auth-login", "params": { "email": "admin@test.com", "password": "secret" } },
+    { "type": "assert_url", "value": "/dashboard" }
+  ]
+}
+```
+
+Module definition (in `e2e/modules/auth-login.json`):
+```json
+{
+  "$module": "auth-login",
+  "description": "Log in with email/password",
+  "params": {
+    "email": { "required": true, "description": "User email" },
+    "password": { "required": true, "description": "User password" }
+  },
+  "actions": [
+    { "type": "goto", "value": "/login" },
+    { "type": "type", "selector": "#email", "value": "{{email}}" },
+    { "type": "type", "selector": "#password", "value": "{{password}}" },
+    { "type": "click", "text": "Sign In" },
+    { "type": "wait", "selector": ".dashboard" }
+  ]
+}
+```
+
+## Suite Naming & Ordering
+
+Files can have numeric prefixes for execution order:
+- `01-auth.json`, `02-dashboard.json`, `03-settings.json`
+
+The `--suite` flag strips the prefix when matching: `--suite auth` finds `01-auth.json`.
+
+## Excluding Tests
+
+Use `exclude` in config to skip files when running `--all`:
+
+```js
+// e2e.config.js
+export default {
+  exclude: ['explore-*', 'debug-*', 'draft-*']
+};
+```
+
+Individual `--suite` runs are not affected by exclude patterns.
+
+## Environment Profiles
+
+Define named profiles in config:
+
+```js
+// e2e.config.js
+export default {
+  baseUrl: 'http://host.docker.internal:3000',
+  environments: {
+    staging: { baseUrl: 'https://staging.example.com' },
+    production: { baseUrl: 'https://example.com', concurrency: 5 }
+  }
+};
+```
+
+Activate with `--env staging` or `E2E_ENV=staging`. Profile values override all other config.
+
+## CI Output Formats
+
+Use `--output <format>` to control report output: `json` (default), `junit` (JUnit XML), or `both`. JUnit XML is generated without external dependencies and saved to `{screenshotsDir}/junit.xml`. The `generateJUnitXML()` function is also exported from the programmatic API.
+
+## Config Priority (ascending)
+
+1. Hardcoded defaults
+2. `e2e.config.js` or `e2e.config.json`
+3. Environment variables (`BASE_URL`, `CONCURRENCY`, etc.)
+4. CLI flags (`--base-url`, `--concurrency`, etc.)
+5. Environment profile merge (via `--env`)