e2e 0.15.1 → 0.16.0

package/docs/debugging.mdx

@@ -0,0 +1,291 @@
+---
+title: Debugging a run
+description: Read the report, open the evidence, watch the browser, and see what the model was sent.
+---
+
+Start with the failure in the terminal. Every run also writes
+`.e2e/report.json`, with links to the screenshots, traces, downloads, and
+recordings captured by the selected engine.
+
+## Read the failure
+
+The `list` reporter's `Failed Tests` section shows the error, failing line,
+and screen at failure. For a failed locator, it also lists nearby matches
+that may explain the problem. The exit code identifies the failure type:
+
+| Exit | Meaning | Retry the job? |
+| ---: | --- | --- |
+| 1 | A test failed | Only if the test is flaky |
+| 2 | Configuration, collection, or policy | Fix the reported problem first |
+| 3 | Engine, app process, model provider, or artifact failure | If the cause is temporary |
+| 4 | An internal runner error | Report it |
+| 130 | The run was interrupted | |
+
+## The failure pages
+
+Use `--reporter list,markdown` to write `.e2e/summary.md` and a page for each
+failed or flaky test under `.e2e/failures/`. A failure page includes the
+source line, steps, recent model turns, and screen text:
+
+````markdown
+# ✗ todos › archives a todo
+
+`tests/todos.e2e.ts` · failed · 16.2s
+
+**ASSERTION_FAILED**
+
+```text
+agent.act failed: the todo exposes only a "Delete" button; no "Archive" button is present.
+```
+
+Look at: `tests/todos.e2e.ts:8`
+Failed the same way on both attempts: **ASSERTION_FAILED** at step 2.
+
+## Steps
+1. ✓ `app.open` "/todos" (35ms) — `tests/todos.e2e.ts:4`
+2. ✗ `agent.act` "Archive the todo" (16.1s, 6 model calls) — **ASSERTION_FAILED** — `tests/todos.e2e.ts:8`
+
+## Agent turns of step 2
+**Turn 2** `tap({"target":"n19"})`
+> Tapped #n19.
+> Screen changes since revision b4 (now revision b6, path /todos, 29 nodes): 4 added, 2 changed.
+> added #n29 button "Delete Groceries"
+
+## Screen at failure
+URL: `http://127.0.0.1:3000/todos`
+```text
+#n18 textbox "New todo"
+#n19 button "Add"
+ #n34 listitem "Groceries" testid="todo"
+  #n37 button "Delete Groceries"
+```
+````
+
+`Look at` points to the failing call. Assertion failures include expected
+and observed values; locator failures include the requested locator and
+nearby matches. Repeated failure details help you compare attempts.
+
+## The report
+
+Use `.e2e/report.json` for scripts and integrations. `--reporter json` prints
+the same document to stdout:
+
+```bash
+jq '.run | {status, exitCode, errors}' .e2e/report.json
+jq '.run.results[] | select(.status != "passed") | {titlePath, file, status}' .e2e/report.json
+jq '.run.results[] | select(.status != "passed") | .attempts[-1]
+    | {status, error, failure, steps: [.steps[] | select(.status != "passed") | {api, label, source, status, error}], artifacts}' .e2e/report.json
+```
+
+Check `run.errors` for run-level failures, then the last attempt of each
+failed result:
+
+- `error` contains the code, message, source line, and assertion details.
+- `failure` identifies the screen and screenshot artifacts and any locator
+  candidates.
+- `steps` shows the calls leading to the failure. Failed agent steps include
+  recent model turns.
+- `artifacts` lists the captured files.
+
+## Artifacts
+
+Artifacts land under `.e2e/artifacts/` (`<output>/artifacts/` when the
+[`output`](/reference/config#output) is moved), one directory per attempt.
+Each run empties it first, so it holds the latest run's evidence:
+
+- **Screenshots.** Failed attempts capture one whenever the engine can and
+  pixels are allowed. The built-in `agent.assert` also captures one after
+  either verdict unless `screenshot: false` disables it. Capture can fail or
+  be withheld. Use `app.screenshot('label')` to capture one explicitly.
+- **Playwright traces** on the attempts the [`trace`](/reference/config#trace)
+  mode records: every attempt locally, the first retry in CI. Open one with
+  `npx playwright show-trace <file>`; `--trace off` skips the cost for one
+  run, `--trace` records every attempt of a CI run.
+- **Downloads** saved through `browser.waitForDownload`, recorded with kind
+  `download`. They are the app's bytes: `redaction: "incomplete"` unless a
+  secret reached the session (filled, or held by the engine for basic auth)
+  and the file is text, which the runner rewrites and marks `complete`.
+- **Videos** and **agent transcripts** when you ask for them; see below.
+
+After a secret fill, no screenshots are taken for the rest of the attempt; see
+[screenshots after a secret fill](/security#screenshots-after-a-secret-fill).
+
+Playwright traces and videos need separate care. Trace redaction replaces
+secret text with `<secret:name>` and drops the trace's screencast frames, so
+the viewer shows actions, DOM snapshots, and network, but no filmstrip. An
+image the app served itself stays, even one that draws the secret. Videos
+mask nothing and are marked `redaction: "incomplete"`.
+
+If trace rewriting fails, the runner deletes the trace from the attempt's
+artifact directory. For a path outside that directory, it withholds the
+artifact with `TRACE_WITHHELD` and leaves the file untouched. See the
+[security model](/security#where-redaction-stops) for all redaction limits.
+
+## Watch it happen
+
+<CodeGroup>
+```bash npm
+npx e2e run --headed                      # a visible browser
+npx e2e run --video                       # record every attempt
+npx e2e run --video=retain-on-failure     # record every attempt, keep the failures
+```
+
+```bash pnpm
+pnpm exec e2e run --headed                      # a visible browser
+pnpm exec e2e run --video                       # record every attempt
+pnpm exec e2e run --video=retain-on-failure     # record every attempt, keep the failures
+```
+
+```bash bun
+bunx e2e run --headed                      # a visible browser
+bunx e2e run --video                       # record every attempt
+bunx e2e run --video=retain-on-failure     # record every attempt, keep the failures
+```
+</CodeGroup>
+
+Each recording lands under its attempt's artifact directory. Playwright
+writes WebM, one file per page the attempt opened; agent-device writes MP4.
+The failure recap prints the path, or the URL of a recording a hosted
+browser keeps. Each recording carries `startedAt`, so a step's `startedAt`
+minus the recording's is where that step is in the video.
+
+To record without the flag, set `video` in the config. `on-first-retry` is
+the cheap CI mode: a test that passes first time records nothing, and a
+flaky one leaves a recording of its retry.
+
+```ts
+export default {
+  video: 'on-first-retry',
+  retries: 1,
+  targets: [
+    { name: 'chromium', engine: web(), app: { url } },
+    { name: 'firefox', engine: web({ browser: 'firefox' }), app: { url }, video: 'retain-on-failure' },
+  ],
+} satisfies E2EConfig;
+```
+
+A target's `video` wins over the config's, `--video` over both. A test's own
+option wins over all of them, which records one flow you are chasing
+without recording the suite. `trace` takes the same modes and follows the
+same order:
+
+```ts
+test('checkout', { video: 'on' }, async ({ agent }) => {
+  await agent.act('Buy the first product in the catalog');
+});
+```
+
+`--video [mode]` takes an optional value, so put test files before it or
+write `--video=<mode>`. The flag and the config's `video` apply to the targets
+whose engine can record, with one notice naming the rest. A mode set on a
+target or a test is required: an engine that cannot record it fails the run
+with `UNSUPPORTED_ARTIFACT` before any test starts.
+[Recording](/reference/config#recording) has every mode and file.
+
+## Console output
+
+`console.log` and `console.error` inside a test print as the test runs, above
+the live view, under a heading that names the stream, the target, the file,
+and the test:
+
+```text
+stdout | web tests/todos.e2e.ts > adds a todo
+hello from the test
+```
+
+Nothing is buffered or dropped: every line appears when it is written, in a
+pipe or a CI log as much as in a terminal. The output is not part of the
+report; use `expect` or an artifact for anything a later reader needs.
+
+## Agent steps
+
+Use these flags to inspect agent behavior:
+
+<CodeGroup>
+```bash npm
+npx e2e run --debug     # timings and a step table on stderr
+npx e2e run --ai-trace  # every model call, for a trace viewer
+npx e2e run --no-cache  # run the agent live instead of replaying
+```
+
+```bash pnpm
+pnpm exec e2e run --debug     # timings and a step table on stderr
+pnpm exec e2e run --ai-trace  # every model call, for a trace viewer
+pnpm exec e2e run --no-cache  # run the agent live instead of replaying
+```
+
+```bash bun
+bunx e2e run --debug     # timings and a step table on stderr
+bunx e2e run --ai-trace  # every model call, for a trace viewer
+bunx e2e run --no-cache  # run the agent live instead of replaying
+```
+</CodeGroup>
+
+`--debug` appends two tables after the reporter's output: phase timings for
+the run, and one row per agent step with its duration, model calls, tokens,
+prompt-cache share, and cost when the provider reports one. Each planned
+step also saves its model transcript as an artifact.
+
+```console
+[e2e debug] agent steps (execution order, model anthropic/claude-sonnet-4.5, total $0.0312)
+  step                                   total   model  observe  action  calls  tokens in/out  cached       cost
+  act "add a todo named groceries"       3.9s    2.8s    410ms   620ms      3  9412/388       61% (5740)   $0.0198
+  assert "the list shows groceries"      1.2s    1.1s     88ms     0ms      1  3120/41        0% (0)       $0.0114
+```
+
+`--ai-trace` writes model requests and responses to `.e2e/ai-trace.json`.
+Read it with [unbox-ai](https://github.com/tester-army/unbox-ai) to inspect
+prompts, tool calls, and token usage:
+
+<CodeGroup>
+```bash npm
+npx unbox-ai runs .e2e/ai-trace.json
+npx unbox-ai summary .e2e/ai-trace.json --run 1
+```
+
+```bash pnpm
+pnpm dlx unbox-ai runs .e2e/ai-trace.json
+pnpm dlx unbox-ai summary .e2e/ai-trace.json --run 1
+```
+
+```bash bun
+bunx unbox-ai runs .e2e/ai-trace.json
+bunx unbox-ai summary .e2e/ai-trace.json --run 1
+```
+</CodeGroup>
+
+Each agent step appears as a run in the AI trace, with one entry per model
+call. It contains the redacted input sent to the model; image bytes are
+replaced with byte counts. Use unbox-ai to inspect it instead of reading the
+large raw JSON file.
+
+Replayed steps make no model call and leave no AI trace. Add `--no-cache`
+when you need to inspect the whole flow or rule out a stale recording. See
+[Caching agent steps](/cache).
+
+## Common errors
+
+| Code | Usual cause | Fix |
+| --- | --- | --- |
+| `APP_UNREACHABLE` | Nothing answers at `app.url`, or `app.command` never became ready; on a device, the message may name the iOS automation runner instead (busy, wedged, or a snapshot it could not present) | Set `app.command.log` and read it; check the port; raise `app.command.startupTimeout`. For the runner, follow the recovery in the message: `npx agent-device daemon stop`, reboot the simulator (see [Testing iOS and Android](/mobile#troubleshooting)) |
+| `APP_ALREADY_RUNNING` | Something already serves `app.url` when the runner wanted to start `app.command` | Stop it, or set `reuseExisting: true` for local runs |
+| `LOCATOR_NOT_FOUND` | Wrong role or name, text not exact, element inside an iframe, page not open | Read the markup for the accessible name; try `exact: false`; `browser.frameLocator` for iframes; `--headed` to look |
+| `LOCATOR_AMBIGUOUS` | Two matches: a hidden duplicate, a repeated label | Add `{ name }`, scope under a container, `filter`, `first()`, or `{ visible: true }` |
+| `ASSERTION_FAILED` | The expectation is wrong, or the state settles after 5 s; for `agent.assert`, the judgment was false | Compare with the screenshot; `{ timeout }` on the matcher; rewrite the question |
+| `ASSERTION_INCONCLUSIVE` | `agent.assert` asked about, or `agent.extract` asked for, something the screen does not show: the value is on another page, still loading, or only in pixels | Open or wait for the right screen first; ask about what is visible; add `vision: true` when the answer is in pixels (the message says so when the judge read the tree alone) |
+| `ACTION_FAILED` | Element covered, disabled, or detached, or an operation timed out | Wait on the right condition with `expect` first; close overlays |
+| `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT` | The agent ran out of actions, model calls, or time | Scope the goal to one step. Raise the agent entry's `maxSteps` or `maxModelCalls` for the budget, the config `timeout` for an `act` step, or the agent's `judgmentTimeout` for `assert`, `waitFor`, and `extract` |
+| `MODEL_UNAVAILABLE` | No model constructed in the config | See [Models](/models) |
+| `MODEL_PROVIDER_FAILED` | Bad key, rate limit, no credits, or every try of one request went 120 s without an answer | Check the key the provider reads and the quota; for a slow local model, a smaller screen (`agents.<name>.maxObservationBytes`) or a faster model |
+
+Every code, its class, and its exit code are in the
+[errors reference](/reference/errors).
+
+<CardGroup cols={2}>
+  <Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-run">
+    Every flag of `e2e run`.
+  </Card>
+  <Card title="Errors" icon="triangle-exclamation" href="/reference/errors">
+    Every error code.
+  </Card>
+</CardGroup>

package/docs/executors.mdx

@@ -0,0 +1,243 @@
+---
+title: Custom executors
+description: Customize the agent loop or implement your own step executor.
+---
+
+Use a custom executor when a `system` prompt and [project tools](/tools) are
+not enough. There are two options:
+
+1. `createToolLoopExecutor` keeps the built-in loop: budgets, repeated-call
+   detection, verdicts, usage accounting, and debug transcripts. Your tools
+   replace the built-in actions; the loop adds only `complete_step`.
+2. `StepExecutor` lets you implement `runStep(ctx)` yourself. A model and
+   the AI SDK are optional; the `e2e` and `e2e/engine` types compile with no
+   `ai` package installed.
+
+Both receive a deadline, redacted observations, and checked action methods.
+The runner records their verdicts. Either goes under `executor` in an agents
+entry, beside the options that still apply: `model`, `judge`, `context`, and
+the budgets. An entry with `executor` rejects `system` and `tools`; pass
+them to `createToolLoopExecutor` instead.
+
+## Keep the loop, change the vocabulary
+
+This example gives the agent two tools: one reads the screen, one taps a
+node by its id.
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { createToolLoopExecutor } from 'e2e/agent';
+import { web } from '@e2e-dev/web';
+import { gateway, tool } from 'ai';
+import { z } from 'zod';
+
+export default {
+  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
+  agents: {
+    default: {
+      model: gateway('openai/gpt-6-luna-fast'),
+      maxModelCalls: 15,
+      executor: createToolLoopExecutor({
+        name: 'tap-only',
+        system: 'Read the screen before acting. Tap controls by their id from the latest read.',
+        buildPrompt: (ctx) => ctx.step.instruction,
+        tools: (ctx, { guard }) => ({
+          read_screen: tool({
+            description: 'Read the screen as text, one node per line as #id role "name"',
+            inputSchema: z.object({}),
+            execute: () => guard(async () => (await ctx.observe()).text),
+          }),
+          tap: tool({
+            description: 'Tap a node by its id from the latest read_screen',
+            inputSchema: z.object({ id: z.string() }),
+            execute: ({ id }) =>
+              guard(async () => {
+                await ctx.actions.tap({ id: id.replace(/^#/, '') });
+                return `Tapped ${id}.`;
+              }),
+          }),
+        }),
+      }),
+    },
+  },
+} satisfies E2EConfig;
+```
+
+The loop reads the entry's `model` unless `createToolLoopExecutor({ model })`
+brings its own, and its model turns per step are the entry's `maxModelCalls`.
+`buildPrompt` returns a string or message history. Use `prepareMessages`
+between turns to shorten the history or return `{ messages, stop }` when
+the step is not progressing.
+
+`guard()` stops new tool calls once the step is concluding and hands a
+failure back to the model as text. `ctx.actions` authorizes, counts, and
+records each action, so these steps replay from the [cache](/cache) like the
+built-in agent's and run on any engine that supports `tap`.
+
+A tool that works outside `ctx.actions`, such as a call to your test API,
+wraps its body in `ctx.budgets.runTool({ name, mutates })` to take an action
+slot and record the outcome. Never call `ctx.actions` or `ctx.observe` inside
+`runTool`: the call queues behind its own operation and deadlocks. Keep tool
+bodies short and support cancellation. The step deadline cannot cancel an
+underlying operation that ignores its signal.
+
+A custom executor can bring its own `model` or `judge`, as
+`createToolLoopExecutor({ model })` does. If the agents entry sets one too,
+they must name the same provider and model, or the config load fails with
+`INVALID_CONFIG`.
+
+## Replace the executor
+
+This model-free executor supports one assertion,
+`agent.assert('the screen says Ready')`. Other instructions return a blocked
+verdict so the executor cannot claim to have checked something it does not
+understand.
+
+```ts title="e2e.config.ts"
+import type { E2EConfig, StepExecutor } from 'e2e';
+import { web } from '@e2e-dev/web';
+
+const scripted: StepExecutor = {
+  name: 'scripted',
+  version: '1',
+  cache: 'off',
+  async runStep(ctx) {
+    if (ctx.step.kind !== 'assert' || ctx.step.instruction !== 'the screen says Ready') {
+      return {
+        status: 'blocked',
+        errorCode: 'AUTOMATION_UNSUPPORTED',
+        summary: 'This executor only checks whether the screen says Ready.',
+      };
+    }
+    const screen = await ctx.observe();
+    const ready = screen.text.includes('Ready');
+    return {
+      status: ready ? 'passed' : 'failed',
+      summary: ready ? 'The screen says Ready.' : 'Ready is absent from the screen.',
+    };
+  },
+};
+
+export default {
+  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
+  agents: { default: { executor: scripted } },
+} satisfies E2EConfig;
+```
+
+The executor receives this context:
+
+| Member | What it is |
+| --- | --- |
+| `ctx.step` | The instruction, params, and `index` in the attempt's timeline. A registered secret value the test spelled into the instruction or a param string reads `<secret:name>`; a `secrets.get()` handle is a placeholder you fill with `typeSecret` |
+| `ctx.observe({ tree, pixels })` | Redacted screen text, with an optional node tree and masked screenshot |
+| `ctx.actions` | Checked and recorded actions: `tap`, `doubleTap`, `longPress`, `secondaryTap`, `hover`, `type`, `typeSecret`, `press`, `select`, `check`, `drag`, `scrollTo`, `scrollUntil`, `upload`, `scroll`, `navigate`, `back`, and the point and keyboard verbs below |
+| `ctx.target` | The target's `name` and `platform`, and `verbs`, the actions its engine supports. Offer only these to the model. |
+| `ctx.pixelsTainted` | True after a [secret fill](/security#screenshots-after-a-secret-fill), when no screenshots are available. A secret an engine option holds (a basic-auth password) is redacted from text only and does not set it |
+| `ctx.signal` | Aborts on cancellation, at the step deadline, or on any other hard stop. Stop promptly when it fires |
+| `ctx.replayedPrefix` | Set when a cache replay ran part of the step before handing it over: the replayed actions and why replay stopped |
+| `ctx.model`, `ctx.providerOptions` | The agent's configured model and `providerOptions`, when set |
+| `ctx.agentContext` | The agent's `context` plus the test's `agentContext` |
+| `ctx.ledger` | Completed steps in the test |
+| `ctx.attempt` | Test and attempt ids, retry index, abort signal, and a `memory` map that is never reported |
+| `ctx.budgets` | Accounting for custom tools and model calls |
+| `ctx.attachTranscript(text)` | Attaches the model transcript to the step. Kept as a `log` artifact under `--debug`, a no-op otherwise; call once, at conclusion |
+| `ctx.attachTurns(turns)` | Keeps a clipped record of the step's model turns (`{ index, calls, outcome }`) on the step record whatever the debug setting; the report shows the last ones under a failed step |
+| `ctx.attachScreenshot(pixels, label)` | Keeps observed pixels as a `screenshot` artifact of the step and resolves with its report id |
+
+`tapAt(point)` and `hoverAt(point)` use a node id when a listed control
+occupies the point and otherwise call the engine's coordinate action.
+`hitTest(point)` finds the listed control without acting. Keyboard-capable
+engines also provide `typeText`, `pressKey`, and `dismissKeyboard` for the
+focused control. `upload(target, paths)` authorizes every path against the
+project root before the engine sees it: outside the project or hidden is
+`POLICY_DENIED`, a missing file `INVALID_ARGUMENT`.
+
+If semantic capture fails, an observation sets `treeUnavailable: true` and
+omits `tree`. It includes a screenshot when policy allows one, even if pixels
+were not requested. Earlier node ids are invalid, and a missing tree does
+not prove a control is absent. Use fresh ids once the tree returns.
+
+Secrets in params arrive as name-only placeholders.
+`ctx.actions.typeSecret(target, name)` fills an authorized field without
+passing the plaintext through your executor.
+
+### What the executor handles
+
+A custom executor handles `act` and `assert` through `runStep`. Both use the
+agent's `maxSteps` and `maxModelCalls` and default to the configured
+`timeout`. On `assert`, `vision` raises `UNSUPPORTED_CAPABILITY`, since your
+executor chooses what it observes; `screenshot` still attaches evidence after
+your verdict.
+
+`waitFor` and `extract` still use the built-in judgments. They require a
+configured model and raise `MODEL_UNAVAILABLE` without one.
+
+### Driving the page yourself
+
+Use `surfaceOf(handle)` from `@e2e-dev/web` to access the current
+attempt's `Page` and `BrowserContext`. Both accessors throw `INVALID_STATE`
+outside an active attempt.
+
+```ts
+import type { E2EConfig, StepExecutor } from 'e2e';
+import { web, surfaceOf } from '@e2e-dev/web';
+
+const engine = web();
+const own: StepExecutor = {
+  name: 'own-tools',
+  version: '1',
+  cache: 'off',
+  async runStep(ctx) {
+    const page = surfaceOf(engine)!.page();   // the page app.open() navigated
+    await ctx.budgets.runTool({ name: 'click', mutates: true }, () =>
+      page.getByRole('button', { name: 'Save' }).click(),
+    );
+    return { status: 'passed', summary: 'saved' };
+  },
+};
+
+export default {
+  targets: [{ engine, app: { url: 'http://localhost:3000' } }],
+  agents: { default: { executor: own } },
+} satisfies E2EConfig;
+```
+
+Direct Playwright calls bypass the runner's action authorization and the
+replay cache. The runner still enforces the step deadline and records the
+verdict, but these calls have no automatic replay or artifacts. Use
+`ctx.budgets` to count custom tool and model calls.
+
+### Replay cache
+
+`cache` on a `StepExecutor` sets how its `act` steps use the
+[replay cache](/cache); `assert` steps are never cached. `'inherit'`, the
+default, follows the configured cache mode. A recorded step replays without
+calling `runStep`. A replay that stops partway calls `runStep` with
+`ctx.replayedPrefix`: continue from the current screen and do not redo the
+replayed actions. What a step does through `ctx.actions` is recorded; a
+mutating tool call leaves a gap the executor fills on replay. Entries are not
+keyed by executor, so an entry one agent recorded replays for another.
+
+`'off'` never replays or records the executor's steps, so every step reaches
+`runStep`. Use it for a brain whose own history or memory is the source of
+truth, or one that must see every step run. The two `StepExecutor` samples on
+this page set it so their code runs on every call.
+
+## The verdict
+
+Return `passed`, `failed`, or `blocked` with a summary. A blocked verdict
+also needs an error code, and a passed one carries none.
+[How agent steps work](/agent-steps#how-a-step-ends) lists the codes and
+resulting exits. `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT`, and `CANCELLED`
+belong to the runner: return one only when that budget is spent, the step
+deadline has passed, or the test was cancelled. Returning one otherwise, or
+a passed verdict with any code, is refused with `MODEL_OUTPUT_INVALID`.
+
+<CardGroup cols={2}>
+  <Card title="agent reference" icon="wand-magic-sparkles" href="/reference/agent">
+    Every method, option, and budget.
+  </Card>
+  <Card title="Writing an engine" icon="plug" href="/writing-an-engine">
+    Add support for another platform.
+  </Card>
+</CardGroup>