@matware/e2e-runner 1.1.0 → 1.2.1

package/skills/e2e-testing/SKILL.md

@@ -0,0 +1,166 @@
+---
+name: e2e-testing
+description: Create, run, and debug JSON-driven E2E browser tests with Chrome pool integration
+---
+
+# E2E Testing with @matware/e2e-runner
+
+## Overview
+
+`@matware/e2e-runner` is a JSON-driven E2E test runner. Tests are defined as JSON files with sequential browser actions — no JavaScript test code. Tests run in parallel against a Chrome pool (browserless/chrome via Docker) using Puppeteer.
+
+**Key capabilities:** 13 MCP tools for running tests, creating test files, capturing screenshots, analyzing network traffic, verifying GitHub/GitLab issues, and querying a learning system for stability insights.
+
+## Prerequisites
+
+Before running any tests, verify the Chrome pool is available:
+
+```
+e2e_pool_status → check "Available: yes" and session capacity
+```
+
+If the pool is not running, the user must start it via CLI (not available via MCP):
+```bash
+npx e2e-runner pool start
+```
+
+## Core Workflow
+
+The standard test execution flow:
+
+1. **Check pool** → `e2e_pool_status` — confirm Chrome pool is ready
+2. **List suites** → `e2e_list` — discover available test files and modules
+3. **Run tests** → `e2e_run` — execute with `all`, `suite`, or `file` parameter
+4. **Interpret results** — check `summary`, `failures`, `narratives`, `networkSummary`
+5. **View screenshots** → `e2e_screenshot` — retrieve error/verification screenshots by `ss:HASH`
+6. **Drill into network** → `e2e_network_logs` — use `runDbId` to inspect requests/responses
+7. **Check learnings** → `e2e_learnings` — query stability trends, flaky tests, error patterns
+
+### Interpreting Run Results
+
+The `e2e_run` response includes:
+
+- **summary**: pass/fail counts, duration, `runDbId` for drill-down
+- **failures**: failed test names with error messages and error screenshot hashes
+- **narratives**: step-by-step human-readable story of each test execution
+- **networkSummary**: per-test request stats (status distribution, slow/failed requests)
+- **verifications**: tests with `expect` field — call `e2e_screenshot` to visually verify
+- **learnings**: stability insights from the learning system (new failures, flaky patterns)
+
+## Creating Tests
+
+### Basic Structure
+
+```json
+[
+  {
+    "name": "login-flow",
+    "actions": [
+      { "type": "goto", "value": "/login" },
+      { "type": "type", "selector": "#email", "value": "user@example.com" },
+      { "type": "type", "selector": "#password", "value": "secret" },
+      { "type": "click", "text": "Sign In" },
+      { "type": "wait", "selector": ".dashboard" },
+      { "type": "assert_url", "value": "/dashboard" }
+    ]
+  }
+]
+```
+
+Use `e2e_create_test` to write test files. Use `e2e_create_module` for reusable action sequences.
+
+### Key Action Patterns
+
+- **Navigation**: `goto` (full page load), `navigate` (SPA-friendly, non-blocking)
+- **Interaction**: `click` (selector or text), `type`/`fill`, `select`, `press`, `hover`, `scroll`
+- **React/MUI**: `type_react` (controlled inputs), `click_option`, `focus_autocomplete`, `click_chip`, `click_regex`
+- **Assertions**: `assert_text` (page-wide), `assert_element_text` (scoped), `assert_url`, `assert_visible`, `assert_not_visible`, `assert_count`, `assert_attribute`, `assert_class`, `assert_input_value`, `assert_matches`
+- **Extraction**: `get_text` (non-assertion, returns element text), `screenshot`
+- **Advanced**: `evaluate` (run JS in browser), `assert_no_network_errors`, `clear_cookies`
+
+### Visual Verification
+
+Add an `expect` field to any test for AI-powered visual verification:
+
+```json
+{
+  "name": "dashboard-loads",
+  "expect": "Should show patient list with at least 3 rows and no error messages",
+  "actions": [...]
+}
+```
+
+After running, call `e2e_screenshot` with each verification hash and judge the screenshot against the description.
+
+### Reusable Modules
+
+Create modules with `e2e_create_module`, reference them in tests:
+
+```json
+{ "$use": "auth-jwt", "params": { "email": "admin@test.com" } }
+```
+
+For complete action type reference, see [action-types.md](references/action-types.md).
+For JSON format details (hooks, serial, retries, modules), see [test-json-format.md](references/test-json-format.md).
+
+## Issue Verification
+
+Turn GitHub/GitLab bug reports into executable tests:
+
+### Prompt Mode (default, no API key needed)
+
+1. `e2e_issue` with issue URL → returns structured prompt with issue details
+2. Analyze the issue and design test actions
+3. `e2e_create_test` → create the test file
+4. `e2e_run` → execute and verify
+
+### Verify Mode (requires ANTHROPIC_API_KEY)
+
+1. `e2e_issue` with `mode: "verify"` → auto-generates tests via Claude API, runs them, reports result
+2. Test failure = bug confirmed, all pass = not reproducible
+
+Supports both UI tests (`testType: "e2e"`) and API tests (`testType: "api"`).
+
+## Debugging & Analysis
+
+### Network Inspection
+
+```
+e2e_network_logs(runDbId)                     → all requests
+e2e_network_logs(runDbId, errorsOnly: true)    → failed requests only
+e2e_network_logs(runDbId, includeBodies: true) → full request/response bodies
+e2e_network_logs(runDbId, urlPattern: "/api/") → filter by URL pattern
+```
+
+### Learning System
+
+```
+e2e_learnings("summary")    → full project overview
+e2e_learnings("flaky")      → flaky test analysis
+e2e_learnings("selectors")  → selector stability
+e2e_learnings("errors")     → recurring error patterns
+e2e_learnings("test:name")  → drill into specific test history
+```
+
+### On-Demand Capture
+
+Use `e2e_capture` to screenshot any URL without running a full test suite. Useful for visual exploration or verifying current state.
+
+### Dashboard
+
+Start/stop the web dashboard with `e2e_dashboard_start` / `e2e_dashboard_stop` for a visual UI at `http://localhost:8484`.
+
+## Important Rules
+
+1. **Always pass `cwd`** — All MCP tools accept `cwd` (the project root). Always pass it so config files and test directories resolve correctly.
+2. **`baseUrl` default is `http://host.docker.internal:3000`** — Chrome runs inside Docker, so it uses `host.docker.internal` to reach the host machine. Override with `baseUrl` if the app runs on a different port.
+3. **Pool management is CLI-only** — `pool start` and `pool stop` are not available via MCP. Only `e2e_pool_status` is an MCP tool.
+4. **`evaluate` is strict** — Returns starting with `FAIL:`/`ERROR:` or returning `false` will fail the test. Prefer granular assertion actions over `evaluate` with inline JS.
+5. **Serial tests** — Mark tests with `"serial": true` if they share mutable state. They run after all parallel tests.
+6. **Action retries** — Use `"retries": N` on individual actions for flaky selectors, or globally via config.
+
+## References
+
+- [Action Types Reference](references/action-types.md) — Complete catalog of 28+ action types with fields and examples
+- [Test JSON Format](references/test-json-format.md) — JSON structure, hooks, serial, retries, modules, exclude patterns, environment profiles
+- [Troubleshooting](references/troubleshooting.md) — Common problems and solutions

package/skills/e2e-testing/references/action-types.md

@@ -0,0 +1,100 @@
+# Action Types Reference
+
+Complete catalog of all action types supported by @matware/e2e-runner.
+
+## Navigation
+
+| Action | Fields | Description |
+|--------|--------|-------------|
+| `goto` | `value` (URL or path) | Full page navigation. Relative paths are prefixed with `baseUrl`. Waits for `domcontentloaded`. |
+| `navigate` | `value` (URL or path) | SPA-friendly navigation. Uses `load` event with a 5s race timeout — won't block if client-side routing doesn't fire `load`. |
+
+## Interaction
+
+| Action | Fields | Description |
+|--------|--------|-------------|
+| `click` | `selector` OR `text` | Click by CSS selector or by visible text content. Text search covers: `button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"], [role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6, dd, dt`. |
+| `type` / `fill` | `selector`, `value` | Triple-clicks to select all, then Backspace to clear, then types with 20ms delay per character. |
+| `select` | `selector`, `value` | Select an `<option>` value in a `<select>` element. |
+| `clear` | `selector` | Triple-click + Backspace to clear an input field. |
+| `press` | `value` (key name) | Press a keyboard key (e.g. `"Enter"`, `"Tab"`, `"Escape"`, `"ArrowDown"`). |
+| `scroll` | `selector` (optional), `value` (optional) | Scroll element into view, or scroll window by Y pixels (default 300). |
+| `hover` | `selector` | Hover over an element. |
+
+## Framework-Aware (React/MUI)
+
+| Action | Fields | Description |
+|--------|--------|-------------|
+| `type_react` | `selector`, `value` | Types into React controlled inputs using native value setter. Dispatches `input` + `change` events so React state updates. Supports `<input>` and `<textarea>`. |
+| `click_regex` | `text` (regex), `selector` (optional), `value` (`"last"` optional) | Click element whose textContent matches regex (case-insensitive). Default: first match. `value: "last"` for last match. `selector` scopes the search. |
+| `click_option` | `text` | Click a `[role="option"]` element by text — for autocomplete/select dropdowns. Waits for option to appear. |
+| `focus_autocomplete` | `text` (label text) | Focus an autocomplete input by label. Supports MUI `.MuiAutocomplete-root` and `[role="combobox"]`. |
+| `click_chip` | `text` | Click a chip/tag element by text. Searches `[class*="Chip"]`, `[class*="chip"]`, `[data-chip]`. |
+
+## Assertions
+
+| Action | Fields | Description |
+|--------|--------|-------------|
+| `assert_text` | `text` | Check entire page body contains text (substring match). |
+| `assert_element_text` | `selector`, `text`, `value` (`"exact"` optional) | Check specific element's `textContent`. Default: substring match. With `value: "exact"`: strict `trim() ===` comparison. |
+| `assert_url` | `value` | Check current URL. Path-only (`/dashboard`) compares pathname. Full URL does substring match. |
+| `assert_visible` | `selector` | Element exists and is visible (`display`, `visibility`, `opacity` checks). |
+| `assert_not_visible` | `selector` | Passes if element doesn't exist OR is hidden. |
+| `assert_count` | `selector`, `value` | Count matching elements. Supports exact (`"5"`) and operators (`">3"`, `">=1"`, `"<10"`, `"<=5"`). |
+| `assert_attribute` | `selector`, `value` (`"attr=expected"` or `"attr"`) | With `=`: checks attribute value. Without: checks attribute existence. |
+| `assert_class` | `selector`, `value` | Checks `classList.contains(value)`. |
+| `assert_input_value` | `selector`, `value` | Checks `element.value.includes(value)` on input/select/textarea. |
+| `assert_matches` | `selector`, `value` (regex) | Tests element's `textContent` against `new RegExp(value)`. |
+| `assert_no_network_errors` | — | Checks accumulated `requestfailed` events during the test. Fails with error details if any exist. |
+
+### Assertion Disambiguation
+
+- **`assert_text`** → searches the **entire page body** (substring)
+- **`assert_element_text`** → checks a **specific element** (substring, or exact with `value: "exact"`)
+- **`assert_matches`** → checks a specific element against a **regex** pattern
+- **`assert_input_value`** → reads the `.value` property (for form fields)
+
+## Extraction & Utility
+
+| Action | Fields | Description |
+|--------|--------|-------------|
+| `get_text` | `selector` | Returns `{ value: textContent.trim() }`. Non-assertion — never fails. |
+| `screenshot` | `value` (filename, optional) | Captures screenshot. Filename gets timestamp suffix for uniqueness. |
+| `wait` | `selector` OR `text` OR `value` (ms) | Wait for selector, text on page, or fixed delay. |
+| `evaluate` | `value` (JS code) | Run JavaScript in browser context. **Strict**: returns starting with `FAIL:`/`ERROR:` → test fails. Returns `false` → test fails. |
+| `clear_cookies` | `value` (origin, optional) | Clears cookies, localStorage, sessionStorage for origin. |
+
+## Action-Level Retry
+
+Any action can have `"retries": N` for per-action retry on failure:
+
+```json
+{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
+{ "type": "wait", "selector": ".lazy-loaded", "retries": 2 }
+```
+
+Delay between retries: `actionRetryDelay` config (default 500ms).
+
+## Examples
+
+### React input + autocomplete flow
+```json
+{ "type": "focus_autocomplete", "text": "Diagnosis" },
+{ "type": "type_react", "selector": "#diagnosis-input", "value": "Cefalea" },
+{ "type": "click_option", "text": "Cefalea tensional" }
+```
+
+### Regex click (last match)
+```json
+{ "type": "click_regex", "text": "start encounter", "selector": "button", "value": "last" }
+```
+
+### Form validation assertions
+```json
+{ "type": "assert_attribute", "selector": "input#email", "value": "type=email" },
+{ "type": "assert_attribute", "selector": "button.submit", "value": "disabled" },
+{ "type": "assert_class", "selector": ".nav-item:first-child", "value": "active" },
+{ "type": "assert_input_value", "selector": "#email", "value": "user@example.com" },
+{ "type": "assert_matches", "selector": ".phone", "value": "\\d{3}-\\d{3}-\\d{4}" },
+{ "type": "assert_count", "selector": ".table-row", "value": ">3" }
+```

package/skills/e2e-testing/references/test-json-format.md

@@ -0,0 +1,159 @@
+# Test JSON Format Reference
+
+## Basic Format (Array)
+
+A test file is a JSON array of test objects:
+
+```json
+[
+  {
+    "name": "test-name",
+    "actions": [
+      { "type": "goto", "value": "/page" },
+      { "type": "assert_text", "text": "Expected content" }
+    ]
+  }
+]
+```
+
+## Object Format (with Hooks)
+
+When hooks are needed, use the object format:
+
+```json
+{
+  "hooks": {
+    "beforeAll": [{ "type": "goto", "value": "/setup" }],
+    "beforeEach": [{ "type": "goto", "value": "/" }],
+    "afterEach": [],
+    "afterAll": []
+  },
+  "tests": [
+    { "name": "test-1", "actions": [...] }
+  ]
+}
+```
+
+**Hook lifecycle:**
+- `beforeAll` — runs once before all tests (on a separate browser page, state does NOT carry over)
+- `beforeEach` — runs before each individual test (on the test's own page)
+- `afterEach` — runs after each test
+- `afterAll` — runs once after all tests
+
+> **Warning**: `beforeAll` runs on a separate page that closes before tests start. Don't use it for browser state setup (cookies, localStorage). Use `beforeEach` instead.
+
+## Test Options
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | string | **Required.** Test identifier. |
+| `actions` | array | **Required.** Sequential browser actions. |
+| `expect` | string | Visual verification description. Triggers auto-screenshot + AI judgment. |
+| `serial` | boolean | Run sequentially after all parallel tests (for shared state). |
+| `retries` | number | Per-test retry count on failure. Overrides global config. |
+| `timeout` | number | Per-test timeout in ms. Overrides global `testTimeout` (default 60000). |
+
+## Serial Tests
+
+Tests that share mutable state should be marked serial to prevent race conditions:
+
+```json
+{ "name": "create-record", "serial": true, "actions": [...] },
+{ "name": "verify-record", "serial": true, "actions": [...] }
+```
+
+Serial tests run one-at-a-time **after** all parallel tests finish.
+
+## Retry Behavior
+
+### Test-level retries
+```json
+{ "name": "flaky-test", "retries": 3, "actions": [...] }
+```
+
+Or globally: `--retries 2` / `retries: 2` in config. Each retry gets its own timeout. Flaky tests (pass after retry) are logged as "flaky".
+
+### Action-level retries
+```json
+{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
+```
+
+Or globally: `--action-retries 2`. Delay between action retries: `actionRetryDelay` (default 500ms).
+
+## Reusable Modules
+
+Create modules with `e2e_create_module`, reference them in tests:
+
+```json
+{
+  "name": "login-test",
+  "actions": [
+    { "$use": "auth-login", "params": { "email": "admin@test.com", "password": "secret" } },
+    { "type": "assert_url", "value": "/dashboard" }
+  ]
+}
+```
+
+Module definition (in `e2e/modules/auth-login.json`):
+```json
+{
+  "$module": "auth-login",
+  "description": "Log in with email/password",
+  "params": {
+    "email": { "required": true, "description": "User email" },
+    "password": { "required": true, "description": "User password" }
+  },
+  "actions": [
+    { "type": "goto", "value": "/login" },
+    { "type": "type", "selector": "#email", "value": "{{email}}" },
+    { "type": "type", "selector": "#password", "value": "{{password}}" },
+    { "type": "click", "text": "Sign In" },
+    { "type": "wait", "selector": ".dashboard" }
+  ]
+}
+```
+
+## Suite Naming & Ordering
+
+Files can have numeric prefixes for execution order:
+- `01-auth.json`, `02-dashboard.json`, `03-settings.json`
+
+The `--suite` flag strips the prefix when matching: `--suite auth` finds `01-auth.json`.
+
+## Excluding Tests
+
+Use `exclude` in config to skip files when running `--all`:
+
+```js
+// e2e.config.js
+export default {
+  exclude: ['explore-*', 'debug-*', 'draft-*']
+};
+```
+
+Individual `--suite` runs are not affected by exclude patterns.
+
+## Environment Profiles
+
+Define named profiles in config:
+
+```js
+// e2e.config.js
+export default {
+  baseUrl: 'http://host.docker.internal:3000',
+  environments: {
+    staging: { baseUrl: 'https://staging.example.com' },
+    production: { baseUrl: 'https://example.com', concurrency: 5 }
+  }
+};
+```
+
+Activate with `--env staging` or `E2E_ENV=staging`. Profile values override all other config.
+
+## Config Priority (ascending)
+
+1. Hardcoded defaults
+2. `e2e.config.js` or `e2e.config.json`
+3. Environment variables (`BASE_URL`, `CONCURRENCY`, etc.)
+4. CLI flags (`--base-url`, `--concurrency`, etc.)
+5. Environment profile merge (via `--env`)

package/skills/e2e-testing/references/troubleshooting.md

@@ -0,0 +1,182 @@
+# Troubleshooting Guide
+
+## Pool Connection Issues
+
+### "Pool not reachable" / Connection refused
+
+**Cause**: Chrome pool (browserless/chrome Docker container) is not running.
+
+**Fix**:
+```bash
+npx e2e-runner pool start
+npx e2e-runner pool status   # verify it's running
+```
+
+Pool management is CLI-only — `pool start` and `pool stop` are not available via MCP.
+
+### "Pool at capacity" / Tests queuing
+
+**Cause**: All Chrome sessions are occupied.
+
+**Fix**: Increase capacity or reduce concurrency:
+```bash
+npx e2e-runner pool stop
+npx e2e-runner pool start --max-sessions 10
+```
+Or reduce test concurrency: `--concurrency 2`
+
+The runner checks `/pressure` before each connection and waits up to 60s for a free slot.
+
+### Docker not running
+
+**Cause**: Docker daemon is not started.
+
+**Fix**: Start Docker Desktop or `sudo systemctl start docker`, then `npx e2e-runner pool start`.
+
+## React / SPA Issues
+
+### React inputs not updating state
+
+**Symptom**: `type` action enters text but React state doesn't change (form validation fails, submit disabled).
+
+**Fix**: Use `type_react` instead of `type` for React controlled inputs:
+```json
+{ "type": "type_react", "selector": "#email", "value": "user@test.com" }
+```
+
+`type_react` uses the native value setter and dispatches `input` + `change` events that React's synthetic event system recognizes.
+
+### SPA navigation not completing
+
+**Symptom**: `goto` hangs or times out on client-side route changes.
+
+**Fix**: Use `navigate` instead of `goto` for SPA route changes:
+```json
+{ "type": "navigate", "value": "/new-page" }
+```
+
+`navigate` uses a 5s race timeout and won't block if `load` doesn't fire (common in SPAs).
+
+### MUI autocomplete not opening
+
+**Symptom**: Clicking or typing in an MUI Autocomplete doesn't open the dropdown.
+
+**Fix**: Use `focus_autocomplete` to properly focus by label text:
+```json
+{ "type": "focus_autocomplete", "text": "Search by name" },
+{ "type": "type_react", "selector": "#autocomplete-input", "value": "search term" },
+{ "type": "click_option", "text": "Desired option" }
+```
+
+## Flaky Tests
+
+### Intermittent failures on dynamic content
+
+**Symptom**: Tests pass sometimes, fail others. Usually timing-related.
+
+**Fixes**:
+1. Add explicit `wait` before assertions:
+   ```json
+   { "type": "wait", "selector": ".data-loaded" },
+   { "type": "assert_text", "text": "Expected content" }
+   ```
+
+2. Use action-level retries for known flaky selectors:
+   ```json
+   { "type": "click", "selector": "#dynamic-btn", "retries": 3 }
+   ```
+
+3. Use test-level retries:
+   ```json
+   { "name": "flaky-test", "retries": 2, "actions": [...] }
+   ```
+
+4. Check the learning system for patterns:
+   ```
+   e2e_learnings("flaky") → identify consistently flaky tests
+   e2e_learnings("selectors") → find unstable selectors
+   ```
+
+### Tests interfering with each other
+
+**Symptom**: Tests pass individually but fail when run together.
+
+**Fix**: Mark tests that share mutable state as `serial`:
+```json
+{ "name": "create-item", "serial": true, "actions": [...] },
+{ "name": "verify-item", "serial": true, "actions": [...] }
+```
+
+## Timeout Issues
+
+### Test timeout (default 60s)
+
+**Fix**: Increase per-test or globally:
+```json
+{ "name": "slow-test", "timeout": 120000, "actions": [...] }
+```
+Or globally: `--test-timeout 120000`
+
+### Action timeout (default 10s)
+
+Each action's `waitForSelector` uses the default timeout. Override per-action:
+```json
+{ "type": "wait", "selector": ".slow-element", "timeout": 30000 }
+```
+Or globally: `--timeout 30000`
+
+## Network Errors
+
+### Tests passing but network requests failing
+
+**Symptom**: Tests pass but `networkSummary` shows failed requests.
+
+**Fix**: Enable strict mode to fail tests with network errors:
+```
+e2e_run({ all: true, failOnNetworkError: true })
+```
+
+Or use `assert_no_network_errors` at specific points:
+```json
+{ "type": "goto", "value": "/api-heavy-page" },
+{ "type": "wait", "selector": ".loaded" },
+{ "type": "assert_no_network_errors" }
+```
+
+### Investigating specific failures
+
+Use network log drill-down:
+```
+e2e_network_logs(runDbId, errorsOnly: true)                    → see all failed requests
+e2e_network_logs(runDbId, urlPattern: "/api/patients")          → filter by URL
+e2e_network_logs(runDbId, testName: "create-patient", includeBodies: true) → full request/response
+```
+
+## Common Mistakes
+
+### Using `beforeAll` for browser state
+
+`beforeAll` runs on a separate page that closes before tests. Use `beforeEach` for state setup.
+
+### Using `evaluate` for simple assertions
+
+Prefer granular assertion actions over `evaluate` with inline JS:
+```json
+// Bad: verbose, error-prone
+{ "type": "evaluate", "value": "if (!document.querySelector('h1').textContent.includes('Dashboard')) throw 'not found'" }
+
+// Good: clear, auto-waits
+{ "type": "assert_element_text", "selector": "h1", "text": "Dashboard" }
+```
+
+### Forgetting `cwd` in MCP calls
+
+All MCP tools need `cwd` to resolve config files and test directories. Always pass the project root.
+
+### Path-only `assert_url`
+
+When checking paths, use path-only format (starts with `/`):
+```json
+{ "type": "assert_url", "value": "/dashboard" }
+```
+This compares against the pathname only, ignoring the `host.docker.internal` origin.