@unotest/web 0.5.0 → 0.6.2

package/guides/agent-integration.md

@@ -0,0 +1,297 @@
+---
+status: stable
+last_updated: 2026-05-14
+---
+
+# Agent integration
+
+`@unotest/web` ships an MCP server with **39 tools**. This manual covers
+how to wire it into common MCP clients and what the agent can do once
+connected.
+
+## MCP client configuration
+
+### Claude Code
+
+`.mcp.json` (repo root) or `~/.claude/.mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "unotest-web": {
+      "command": "npx",
+      "args": ["@unotest/web", "mcp"]
+    }
+  }
+}
+```
+
+Claude Code prompts for approval on first start. After approve, the
+39 `mcp__unotest-web__*` tools appear in the agent's tool surface.
+
+### Claude Desktop
+
+`~/Library/Application Support/Claude/claude_desktop_config.json` (macOS):
+
+```json
+{
+  "mcpServers": {
+    "unotest-web": {
+      "command": "npx",
+      "args": ["@unotest/web", "mcp"]
+    }
+  }
+}
+```
+
+### Cursor
+
+Cursor's MCP config follows the same shape — see Cursor docs for the
+file location.
+
+### Verify manually
+
+```sh
+echo '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' \
+  | npx @unotest/web mcp
+```
+
+Should return JSON with `tools: [...39 entries]`.
+
+## Tool catalog
+
+### Core browser actions (22)
+
+Direct equivalents of the DSL surface — usable when the agent wants to
+drive the page outside a `run_test` (e.g., exploratory navigation,
+manual setup).
+
+`new_context`, `close_context`, `goto`, `reload`, `get_url`, `get_title`,
+`click`, `double_click`, `fill`, `press`, `check`, `uncheck`,
+`select_option`, `hover`, `scroll_into_view`, `wait_for`, `wait_for_url`,
+`wait_for_text`, `get_page_snapshot`, `get_frame_snapshot`,
+`get_aria_snapshot`, `find_element`.
+
+`get_page_snapshot` / `get_frame_snapshot` return a compact outline of
+**interactive** elements (action surface). `get_aria_snapshot` returns
+a YAML ARIA tree of the page including **non-interactive** named nodes
+(headings, regions, lists, paragraphs, dialogs, images) plus free text
+— the verification surface. Refs are shared: the same `[ref=eN]`
+resolves the same element across all three tools. Format mirrors
+Playwright `ariaSnapshot` AI mode so agents trained on playwright-mcp
+output need no retraining.
+
+`find_element({role, name?, near?, nth?})` — targeted ARIA search.
+Returns matching element refs without dumping a full snapshot — the
+escape hatch when `get_page_snapshot` / `get_aria_snapshot` exceed
+the MCP token cap (table-heavy fleet UIs with 100+ rows). Refs are
+shared with outline / aria-snapshot.
+
+### Debugger (6)
+
+The reason most agents pick this package over driving Playwright
+directly. Detailed workflow in `debugger-tools.md`.
+
+- `run_test` — load `unotest/e2e/<name>.js`, start an AST execution.
+  Modes: `auto` (run to completion / failure), `step` (pause after every
+  statement). `pauseOnFailure` defaults to `true`.
+- `step` — advance one AST node. Returns the next event.
+- `resume` — run until next pause / completion / failure.
+- `inspect_runtime` — snapshot of runtime state: current line, last
+  event, last failure (with line/col + error), and all local variables.
+- `abort_runtime` — kill the runtime, release the browser context.
+- `list_runtimes` — recover a lost `runtimeId`.
+
+### Multi-context (4)
+
+- `list_pages` — open tabs/windows in the active context
+- `list_frames` — iframe tree
+- `get_active_context` — current page index + URL + frame depth
+- `switch_page` — change active tab
+
+#### Popup auto-switch (recording only)
+
+In `explore_step` the MCP server watches for tabs/popups opened by the
+action during a short grace window. If one appears, the server:
+
+1. Auto-switches the active tab to it (last-opened wins on multi-popup).
+2. Appends a synthetic `set_page` entry to the recording log so the
+   generated DSL contains `setPage(N);` and replays correctly in
+   `run_test`. If a frame scope was active before the switch, synthetic
+   `exit_frame` entries are appended first (frame stack does not survive
+   tab changes).
+3. Adds `autoSwitched: { from, to, url, totalNewPages }` to the
+   `explore_step` JSON response so the agent's next snapshot targets the
+   new tab without confusion.
+
+If popups appeared but every one closed before the switch could land,
+the response carries `autoSwitchAttempted: { collected, allClosed: true }`
+instead.
+
+Tuning via `unotest/.env`:
+
+```
+# Max grace window (ms) — popup-watcher early-exits ~200 ms after the
+# first popup event lands, so the full cap is only paid when no popup
+# arrives. 0 disables. Default 2000. Range [0, 10000].
+UNOTEST_POPUP_GRACE_MS=2000
+```
+
+Auto-switch runs **only** under `explore_step` (recording / ad-hoc).
+`run_test` stays strict — if the author omits `setPage`, the test fails.
+
+### Failure bundle introspection (6)
+
+- `list_failures` — bundles on disk, newest first
+- `get_failure_trace` — `failure.json` + error + line/col
+- `get_failure_console` — captured console entries (last 100)
+- `get_failure_a11y` — semantic-DOM snapshot at failure moment
+- `get_failure_screenshot` — PNG (returned as base64; tool returns the
+  path otherwise)
+- `get_failure_network` — HAR (tier 3 — not yet wired, see
+  `KNOWN_ISSUES.md`)
+
+### `agent_fix` (1)
+
+Build a structured fix-context for a failed run (D-25). Composes the
+last failure + relevant DOM + scenario excerpt + suggested action
+classification. **Does not call an LLM** — the agent receiving the
+bundle decides what to do. Detailed in `agent-fix.md`.
+
+### `get_last_mcp_log` (1)
+
+Read the server's own diagnostic log instead of guessing from a
+truncated error. Gated on `UNOTEST_DEBUG` (see [Debug logging](#debug-logging));
+when off it returns an actionable hint. No args → the full last tool-call
+with its nested resolver / driver / snapshot events. `filter:"errors"` →
+the last failed call plus recent failing exits. `filter:"layer:resolver"`
+(or driver / snapshot / refs / tool) → recent events from that layer.
+`filter:"tool:explore_step"` → recent events for that tool. `limit` caps
+the count (default 50).
+
+## Debug logging
+
+Set `UNOTEST_DEBUG=1` in `unotest/.env` and restart the MCP server to
+record a structured **JSONL** trace plus **per-call folders with
+ground-truth artifacts** under `unotest/.debug/`:
+
+- `tool` — each MCP call: enter (args) + exit (ok / error / timing).
+- `resolver` — ref → stable-locator resolution: the element hint, the
+  ranked candidates, and each attempt's match count + verdict.
+- `driver` — Playwright actions: op, locator, timeout, and on failure the
+  tail of Playwright's call log (what intercepted the click, etc.).
+- `snapshot` — every `get_*_snapshot` / `find_element`: byte size, ref
+  count, timing.
+- `refs` — ref totals + counter ceiling per snapshot.
+
+Every event carries `trace` (the explorationId, or the callId for ad-hoc
+calls), `span` (the callId), and — for ref locators — `sourceSnapshot`, the
+snapshot that minted the ref (its provenance / "before" state).
+
+**Ground-truth artifacts** answer "did the snapshot glitch?" and "did the
+locator hit the right element?" without guessing — each tool call gets a
+folder:
+
+```
+unotest/.debug/
+  <pid>-<isoStart>/                  # one folder per process (session)
+    log.jsonl                        # the event stream (rotation-aware)
+    index.jsonl                      # one line per completed tool call
+    <NNN>-<tool>/                    # one folder per tool call, seq-ordered
+      call.json                      # args + nested events + result
+      before.html  after.html        # full DOM before/after an action (read-only)
+      before.txt   after.txt         # readable outline before/after (read-only)
+      diff.txt                       # line diff before.txt↔after.txt — what changed
+      aria.yaml | outline.txt | find_element.json    # the FULL snapshot, rendered
+      resolved/<ref>.html            # outerHTML of what the locator matched
+      page.html  screenshot.png      # failure forensic
+  mcp-latest.jsonl                   # symlink to the active log (tail -f)
+  last-call.json                     # the last tool call + its nested events
+  last-error.json                    # the last FAILED tool call
+```
+
+For every action (`explore_step` / `explore_record`) the proxy writes a
+**read-only** `before.*`/`after.*` pair — `before.html`/`after.html` (full
+DOM, `data-unotest-ref` baked in) and `before.txt`/`after.txt` (readable
+outline) plus `diff.txt` (a line diff of the two outlines — read this first:
+it shows at a glance what the action changed, e.g. `value=""` → `value="…"`,
+303 rows appearing, or `(no change)` for a no-op). Unconditional (success,
+no-op, or failure), so you can always see what the action changed and check
+whether a ref was even on the page. These captures never mint/sweep refs, so
+they can't change behaviour.
+
+The rendered snapshot (`aria.yaml` / `outline.txt` — the same text the tool
+returns) is the verification surface, never truncated. `resolved/<ref>.html`,
+`page.html`, and `before/after.html` are full DOM (not capped); the
+`UNOTEST_DEBUG_HTML_MAX_BYTES` (64 KB) cap applies to the failure-forensic
+`page.html` only.
+Old call folders are pruned beyond `UNOTEST_DEBUG_KEEP_CALLS` (200).
+
+Read it from inside the agent with `get_last_mcp_log` (`filter:"network"`
+and `filter:"exploration:<id>"` also work), or `tail -f
+unotest/.debug/mcp-latest.jsonl` / `jq` from a shell. `UNOTEST_DEBUG=summary`
+keeps only the `tool` + `snapshot` layers and writes no artifacts. Off by
+default — zero I/O when unset.
+
+> Network capture (a 6th `network` layer for failed / 4xx-5xx / navigation
+> events) is planned; `filter:"network"` is accepted but currently empty.
+
+### Auditing a run against the rules — `audit_last_run`
+
+`audit_last_run` runs a deterministic rubric (no LLM) over the recorded
+trajectory and reports where the testing rules were broken — a self-check
+the agent can run after a flow:
+
+- **violation** — something to fix in how the test was built: a ref acted on
+  with no preceding snapshot (stale/guessed), the same ref failing to
+  resolve 3+ times (stuck), a recorded step missing its `section`/`description`.
+- **report** — a signal you should see but didn't necessarily cause: a
+  locator forced to fall back to brittle text/CSS (no testid/role+name), or
+  an `allowNoRef` escape. (Whether an action actually changed anything is
+  read off the `before.*`/`after.*` captures, not inferred.)
+
+Pass `exploration:"<id>"` to scope to one recording session; omit to audit
+the whole process log.
+
+**PII warning.** Tool arguments are logged **verbatim**, so a
+`fill(value: "<password>")` lands in the log. It's a local, gitignored
+file that is never uploaded — but if that's a concern, keep `UNOTEST_DEBUG`
+off. No redaction is applied.
+
+## Recommended system prompt
+
+The shipped agent prompt is in
+`src/mcp/prompts/agent-test-author.md`.
+Two-line summary for your custom system prompt if you want to integrate
+without copy-paste:
+
+> When writing or debugging `@unotest/web` scenarios, prefer
+> `getByTestId` → `getByRole(name)` → `getByLabel` → `getByText` →
+> `locator(css)`. When a scenario fails, call `inspect_runtime` for
+> local-variable state before calling `step` / `resume`. Prefer typed
+> getters (`getInnerText`, `getAttribute`) over `evaluate(js)`.
+
+## Capability model
+
+The MCP server only acts on the consumer project it was launched from:
+its cwd anchors `unotest/`, `unotest.config.*`, `.unotest/failures/`,
+and the sandbox primitives.
+
+What the server **cannot** do (by design — D-25):
+
+- Call external LLM APIs
+- Hold or load API keys
+- Apply patches to your scenarios or source code automatically
+- Access files outside the project root
+
+What it **can** do (with your approval):
+
+- Run your scenarios + drive browsers
+- Read failure artifacts from `.unotest/failures/`
+- Execute sandbox primitives — `shell(cmd, …args)` (execFile, not
+  shell), `dbQuery` / `dbExec` (against `sandbox.database`), `apiCall`
+  (path-only against `sandbox.apiBaseUrl`)
+
+Pinning `sandbox.*` in `unotest.config.*` is how you constrain what the
+agent's scenarios can touch: a scenario cannot redirect `shell` cwd or
+hit an arbitrary HTTP host.