e2e 0.15.2 → 0.17.0

package/docs/models.mdx

@@ -3,8 +3,12 @@ title: Models
 description: Configure a model with your own subscription, an API key, or a local server.
 ---
 
+import { ModelProvider } from './snippets/model-provider.jsx';
+
 Agent steps use the model configured in `e2e.config.ts`. You can use an
 [existing subscription](/subscriptions), an API key, or a local server.
+e2e uses the [AI SDK](https://ai-sdk.dev/providers). You can use
+[all AI SDK providers that support tool calls](#ai-sdk-providers).
 `e2e init` offers these choices and writes the config for you. For the first
 setup, follow the [quickstart](/quickstart#use-a-subscription-or-api-key).
 
@@ -77,7 +81,8 @@ const agent = { model: openai('gpt-6-luna') };
 ```
 
 Use `agent` as `agents.default`. Other AI SDK providers work the same way,
-with their own package, model IDs, and authentication settings.
+with their own package, model IDs, and authentication settings. See
+[AI SDK providers](#ai-sdk-providers).
   </Tab>
   <Tab title="Local or self-hosted">
 
@@ -103,11 +108,6 @@ Use `agent` as `agents.default`. Add `apiKey` if the server requires one.
 Model IDs depend on the provider. Gateways usually use `provider/model`;
 direct providers use their own model IDs.
 
-DeepSeek's V4 models (`@ai-sdk/deepseek`) think by default, and in that mode
-they refuse the forced tool choice agent steps ask for. The step is sent again
-with `auto` and an instruction to reply with tool calls only, and later steps
-on that model start that way, so thinking can stay on.
-
 ### Vercel OIDC instead of a key
 
 Without `AI_GATEWAY_API_KEY`, `gateway()` authenticates with a Vercel OIDC
@@ -145,6 +145,46 @@ An OIDC failure is the same `MODEL_PROVIDER_FAILED` as a bad key, and the AI
 SDK's message names only `AI_GATEWAY_API_KEY`. With no key set, check that the
 Vercel CLI is signed in and that `.vercel/project.json` exists.
 
+## AI SDK providers
+
+Select a provider. The three steps below show the setup for that provider.
+
+<ModelProvider show="select" />
+
+The list shows the AI SDK providers that support tool calls. A provider that
+is not in the list also works if it uses the AI SDK language model
+specification v2 or later. Its model must support tool calls and images.
+
+### Install the provider package
+
+Install `ai` and the provider package.
+
+<ModelProvider show="install" />
+
+### Set the credentials
+
+The provider reads its credentials from the environment. e2e does not load
+`.env` files. Export the variables in your shell or CI, or load the file in
+the config with [`process.loadEnvFile()`](/reference/config#loading).
+
+<ModelProvider show="env" />
+
+If a key is missing or wrong, the first agent step fails with
+`MODEL_PROVIDER_FAILED`. Some providers read the key when the config loads.
+For these providers, a missing key stops all runs with `CONFIG_LOAD_FAILED`.
+
+### Add the model to the config
+
+Set the model in `agents.default`. The model ID is an example. Use a model
+that supports tool calls and images.
+
+<ModelProvider show="config" />
+
+DeepSeek's V4 models (`@ai-sdk/deepseek`) think by default, and in that mode
+they refuse the forced tool choice agent steps ask for. The step is sent again
+with `auto` and an instruction to reply with tool calls only, and later steps
+on that model start that way, so thinking can stay on.
+
 ## Several models in one config
 
 Give agents different names to use different models:
@@ -198,6 +238,12 @@ Model input is the redacted screen, the step instruction, and the
 screenshots the agent asks for. e2e reads nothing back from the provider
 after a response, so a run needs no state kept on the provider's side.
 
+Every model request names e2e: the `User-Agent` starts with
+`e2e/<version> (<platform>; <arch>)`, and `HTTP-Referer:
+https://tester.army/e2e` with `X-Title: e2e` attribute the traffic to e2e on
+the Vercel AI Gateway and OpenRouter. These replace the same headers set on
+the provider instance.
+
 **OpenAI and Azure OpenAI.** Every request carries `store: false` and a
 prompt cache key. The provider keeps no copy of the conversation, a
 reasoning model's earlier turns travel inline as encrypted content, and the
@@ -233,7 +279,7 @@ organizations with zero data retention.
 
 <CardGroup cols={2}>
   <Card title="Subscriptions" icon="key" href="/subscriptions">
-    Sign in with ChatGPT, Copilot, or SuperGrok.
+    Sign in with ChatGPT, Copilot, OpenCode Console, or SuperGrok.
   </Card>
   <Card title="Agents and personas" icon="users" href="/agents">
     Choose agents for tests and signed-in users.

package/docs/physical-devices.mdx

@@ -0,0 +1,76 @@
+---
+title: Physical devices
+description: Run mobile tests on an iPhone or Android phone connected to your machine.
+---
+
+agent-device drives a phone plugged into your machine the same way it
+drives a simulator, so a physical device needs no other engine or option.
+List what agent-device sees, then name the phone in `device` as it is
+listed:
+
+<CodeGroup>
+```bash npm
+npx agent-device devices --platform ios
+```
+
+```bash pnpm
+pnpm exec agent-device devices --platform ios
+```
+
+```bash bun
+bunx agent-device devices --platform ios
+```
+</CodeGroup>
+
+```ts
+mobile({ platform: 'ios', device: 'QA iPhone' })
+```
+
+Name a phone by its name, not its id. The engine reads a `device` string as
+an id only when it is a simulator UDID or an `emulator-<port>` serial, so a
+phone's UDID or adb serial is looked up as a name and not found. The first
+device with a matching name wins, so give each phone a unique name: two
+phones both called `iPhone` or `Pixel 8` would bind one phone to two
+workers. Without `device`, a connected phone counts as booted and joins the
+pool beside any booted simulators, so name it to be sure a run lands on it.
+
+Set the phone up once:
+
+- **iOS**: the phone is paired and trusted, unlocked, with Developer Mode
+  on. agent-device builds and signs its XCTest runner onto the phone, so set
+  `AGENT_DEVICE_IOS_TEAM_ID` to your team id and `AGENT_DEVICE_IOS_BUNDLE_ID`
+  to a unique reverse-DNS id (`com.example.agentdevice.runner`); the
+  signing profile must allow that id and `<id>.uitests`. Set them before the
+  first agent-device command, or run `npx agent-device daemon stop` after:
+  a running daemon keeps the environment it started with. The first run
+  builds the runner and takes longer. `app.appPath` must be a signed device
+  build: a simulator `.app` does not install on a phone.
+- **Android**: USB debugging is on and the phone is authorized, listed as
+  `device` in `adb devices`. `app.appPath` is the same `.apk`.
+
+`npx agent-device doctor` checks the setup, and
+`npx agent-device help physical-device` covers iOS signing in detail.
+
+Some device controls need a simulator or emulator, and fail on a phone with
+agent-device's message:
+
+- On an iPhone, every device setting is simulator-only: `app.permissions`
+  and `device.setPermission`, `app.clearState()`, `device.setNetwork`,
+  `device.setAirplaneMode`, `device.setLocation`, `device.clearLocation`,
+  `device.setAppearance`, the biometric methods, and `device.clearKeychain`.
+  So are `device.clipboard`, `device.setClipboard`, and `device.fold`. For a
+  fresh start, reinstall with `device.installApp(path, { reinstall: true })`.
+- An older iPhone that `xcrun devicectl list devices` does not list is
+  driven without CoreDevice: taps, typing, and the screen work, but
+  installs, `--video`, `device.openLink`, and `app.launchArguments` fail.
+  Install the build from Xcode and pin `app.bundleId`.
+- On an Android phone, `device.setLocation` is emulator-only, and
+  `device.setBiometrics('fingerprint', ...)` works only where the phone
+  exposes `cmd fingerprint`.
+- `--video` on an iPhone is built from screenshots at about 15 frames a
+  second, and drops frames.
+- A phone is not this machine, so `localhost` in the app is the phone
+  itself. Point the app at your machine's network address, or on Android
+  forward the port with `adb reverse tcp:<port> tcp:<port>`.
+
+To run on phones in a device farm, see [Hosted devices](/mobile#hosted-devices).

package/docs/quickstart.mdx

@@ -10,7 +10,7 @@ export const setupPrompt = `Set up e2e end-to-end tests in this project. Docs: \
 1. Run \`npx e2e init --yes\` (or the pnpm or bun equivalent for this project). It writes a web config using Vercel AI Gateway, an example test, the e2e skill, and MCP config. It installs nothing yet.
 2. Read the e2e skill (\`npx e2e guide\` prints it, \`npx e2e guide <topic>\` prints a topic) and follow it for every step below.
 3. Point the target at this app. For a web app, set its URL and the command that starts its dev server. For an iOS or Android app, replace @e2e-dev/web and playwright with @e2e-dev/mobile and rewrite the example test for the mobile engine.
-4. Ask me which model to use for agent steps: a ChatGPT, GitHub Copilot, or SuperGrok subscription, an API key, or a local model. Configure it with its provider package. If it needs a sign-in, give me the \`npx e2e login\` command and wait until I confirm.
+4. Ask me which model to use for agent steps: a ChatGPT, GitHub Copilot, OpenCode Console, or SuperGrok subscription, an API key, or a local model. Configure it with its provider package. If it needs a sign-in, give me the \`npx e2e login\` command and wait until I confirm.
 5. Install dependencies, run the example test, and fix setup errors until it passes.
 6. Write one test for the most important user flow in this app, run it, and iterate until it passes.`;
 
@@ -159,6 +159,7 @@ Connect the provider you chose in the wizard.
   | --- | --- |
   | ChatGPT Plus or Pro | `npx e2e login openai` |
   | GitHub Copilot | `npx e2e login github-copilot` |
+  | OpenCode Console (OpenCode Zen and OpenCode Go) | `npx e2e login opencode-console` |
   | SuperGrok or X Premium+ | `npx e2e login spacexai` |
 
   Copilot reuses your GitHub CLI login if available. See
@@ -245,8 +246,8 @@ to watch a web test, `--no-cache` to skip replaying cached actions.
 ## Next
 
 <CardGroup cols={2}>
-  <Card title="Writing tests" icon="pen-to-square" href="/writing-tests">
-    Write goals, check outcomes, and extract data.
+  <Card title="Goals" icon="pen-to-square" href="/goals">
+    Write test steps as goals in natural language.
   </Card>
   <Card title="Signing in" icon="key" href="/authentication">
     Reuse an app login and keep passwords out of model input.

package/docs/reference/app.mdx

@@ -43,7 +43,7 @@ timeout. They do not accept per-call timeout options.
 
 | Method | Timeout | Throws |
 | --- | --- | --- |
-| `open(path?)` | `config.timeout`, 120000 ms | `APP_URL_REQUIRED` for a path when the target declares no `app.url`, `POLICY_DENIED` for a `file:`, `data:`, or `javascript:` URL, `APP_UNREACHABLE`, `ACTION_FAILED`, `UNSUPPORTED_CAPABILITY` without `session.open` on a target with a URL or without `session.restart` on one without (a device launches its pinned app) |
+| `open(path?)` | `config.timeout`, 120000 ms | `APP_URL_REQUIRED` for a path when the target declares no `app.url`, `POLICY_DENIED` for a URL whose scheme is not `http:` or `https:` (`about:blank` aside), `APP_UNREACHABLE`, `ACTION_FAILED`, `UNSUPPORTED_CAPABILITY` without `session.open` on a target with a URL or without `session.restart` on one without (a device launches its pinned app) |
 | `restart()` | `config.timeout`, 120000 ms | `ACTION_FAILED`, `ENGINE_FAILURE`, `UNSUPPORTED_CAPABILITY` without `session.restart` |
 | `clearState()` | `config.timeout`, 120000 ms | `ACTION_FAILED`, `ENGINE_FAILURE`, `UNSUPPORTED_CAPABILITY` without `session.reset` |
 | `back()` | `config.actionTimeout`, 30000 ms | `APP_NOT_OPEN`, `ACTION_FAILED`, `UNSUPPORTED_CAPABILITY` without `session.back` |

package/docs/reference/cli.mdx

@@ -96,9 +96,9 @@ For a new project, the wizard asks you to choose:
   **Mobile (iOS/Android)** adds `@e2e-dev/mobile`, defaults to iOS on
   macOS and Android elsewhere, and opens Settings. **None** adds nothing.
 - A model provider or subscription for agent steps. Choose Vercel AI Gateway,
-  OpenRouter, an OpenAI-compatible endpoint, ChatGPT, GitHub Copilot, or
-  SuperGrok. Choose None for tests that only use locators. The wizard adds
-  the required packages and model config. Subscription users then sign in
+  OpenRouter, an OpenAI-compatible endpoint, ChatGPT, GitHub Copilot,
+  OpenCode Console, or SuperGrok. Choose None for tests that only use
+  locators. The wizard adds the required packages and model config. Subscription users then sign in
   with `e2e login`. See [Models](/models) and [Subscriptions](/subscriptions).
 - Where to install the agent skill (see [e2e guide](#e2e-guide)):
   `.agents/skills/` for Codex, Cursor, Copilot, Gemini CLI, OpenCode, and
@@ -164,6 +164,7 @@ npx e2e login openai              # ChatGPT, browser sign-in
 npx e2e login openai --device     # sign in with a device code
 npx e2e login github-copilot --from-gh  # use your GitHub CLI login
 npx e2e login github-copilot --client-id <id> [--enterprise-url <host>]
+npx e2e login opencode-console    # OpenCode Console, device code
 npx e2e login spacexai            # SuperGrok, device code
 npx e2e login                     # choose a provider
 ```
@@ -173,6 +174,7 @@ pnpm exec e2e login openai              # ChatGPT, browser sign-in
 pnpm exec e2e login openai --device     # sign in with a device code
 pnpm exec e2e login github-copilot --from-gh  # use your GitHub CLI login
 pnpm exec e2e login github-copilot --client-id <id> [--enterprise-url <host>]
+pnpm exec e2e login opencode-console    # OpenCode Console, device code
 pnpm exec e2e login spacexai            # SuperGrok, device code
 pnpm exec e2e login                     # choose a provider
 ```
@@ -182,6 +184,7 @@ bunx e2e login openai              # ChatGPT, browser sign-in
 bunx e2e login openai --device     # sign in with a device code
 bunx e2e login github-copilot --from-gh  # use your GitHub CLI login
 bunx e2e login github-copilot --client-id <id> [--enterprise-url <host>]
+bunx e2e login opencode-console    # OpenCode Console, device code
 bunx e2e login spacexai            # SuperGrok, device code
 bunx e2e login                     # choose a provider
 ```
@@ -194,7 +197,7 @@ configuration.
 
 | Argument or flag | Type | Default | Behavior |
 | --- | --- | --- | --- |
-| `[provider]` | `openai`, `github-copilot`, or `spacexai` | a picker | The subscription to sign in to |
+| `[provider]` | `openai`, `github-copilot`, `opencode-console`, or `spacexai` | a picker | The subscription to sign in to |
 | `--device` | boolean | `false` | ChatGPT: show a code to enter on another device instead of opening a browser |
 | `--from-gh` | boolean | `false` | GitHub Copilot: reuse the token of the signed-in GitHub CLI |
 | `--client-id <id>` | string | none | GitHub Copilot: the client id of your GitHub OAuth App with the device flow enabled |
@@ -239,8 +242,11 @@ bunx e2e models openai    # one provider
 </CodeGroup>
 
 Asks the vendor which models the stored login serves and prints one line per
-model: the id `chatgpt()`, `copilot()`, or `grok()` takes, the vendor's name
-for it, and its notes (reasoning levels, vision, preview, hidden). Without a
+model: the id `chatgpt()`, `copilot()`, `opencodeConsole()`, or `grok()`
+takes, the vendor's name for it, and its notes (reasoning levels, vision,
+preview, hidden, for Copilot `not enabled` or `no chat or responses` when
+`copilot()` cannot call it, and for OpenCode Console the plan, Zen or Go,
+that serves it). Without a
 stored login, or when the named provider has none, exits 1 with the sign-in
 command; when one of several logins fails to list, the others still print and
 the command exits 1. An unknown provider exits 2. See
@@ -357,7 +363,7 @@ or every test was filtered out.
 | `--exclude-tag <tags>` | comma-separated or repeated | none | Leaves out every test carrying any of the tags, whatever `--tag` or a positional selected. A serial group still runs whole when any member is selected; tag the serial `describe` to exclude the flow. |
 | `--grep <pattern>` | regular expression, repeatable | none | Only tests whose title matches one of the patterns. The text matched is the describe titles and the test title joined by one space (`checkout pays`); the file and the tags are not part of it. A bare pattern is case-sensitive; `'/pattern/i'` adds flags. A pattern that does not compile is a usage error, exit 2. |
 | `--grep-invert <pattern>` | regular expression, repeatable | none | Leaves out every test whose title matches one of the patterns. Same text and pattern forms as `--grep`; both together keep the tests `--grep` matches minus the ones `--grep-invert` matches. |
-| `--last-failed` | boolean | `false` | Only the tests the previous run did not pass, read from the `report.json` this run overwrites, `<output>/report.json` (`--output`, else the config's [`output`](/reference/config#output)): failed, timed out, interrupted, or skipped because a setup, a serial predecessor, a hook, or the worker failed, and every test it selected in the describe or file, on that target, of a `beforeAll` or `afterAll` that failed (for a setup, the tests that consume its session). A rerun that runs the hook again reports it again if it still fails; another filter that leaves its tests out leaves the failure out of the next report too. A test is named per target and agent. No report is `NO_LAST_RUN`; a previous run with nothing failed selects nothing, `NO_TESTS` unless `--pass-with-no-tests`. |
+| `--last-failed` | boolean | `false` | Only the tests the previous run did not pass, read from the `report.json` this run overwrites, `<output>/report.json` (`--output`, else the config's [`output`](/reference/config#output)): failed, timed out, interrupted, or skipped because a setup, a serial predecessor, a hook, or the worker failed or `--max-failures` stopped the run first, and every test it selected in the describe or file, on that target, of a `beforeAll` or `afterAll` that failed (for a setup, the tests that consume its session). A rerun that runs the hook again reports it again if it still fails, and a rerun that runs any test of its scope without it failing resolves it. A test another filter (`--grep`, `--tag`, `--target`, `--agent`, a positional) leaves out of the rerun stays owed: the rerun's report carries its last result, and the hook failure whose scope it is in, under `run.carried`, the next `--last-failed` selects it again, and the [`@e2e-dev/github`](/github) page stays red until it passes (⏹️ for a test the run before interrupted). The rerun's own exit code and summary count only what it ran. It keeps the artifacts the previous report names and writes its own under `artifacts/rerun-<n>/`, so no attempt overwrites an earlier one's files. A test is named per target and agent. No report is `NO_LAST_RUN`; a previous run with nothing failed selects nothing, `NO_TESTS` unless `--pass-with-no-tests`. |
 | `--shard <index/total>` | `2/3` | none | Runs one contiguous slice of the selected tests, cut once every other filter applied. Slices differ in size by at most one test, a serial group counts as one and is never split, and each shard brings only the setup tests its own tests need. Every shard runs the same command with a different index; the order is the report order, so the same suite shards the same way on every machine. A shard that gets no test is `NO_TESTS` unless `--pass-with-no-tests`. See [Sharding](/ci#sharding). |
 | `--headed` | boolean | `false` | Visible UI when the engine supports it. |
 | `--agent <names>` | comma-separated or repeated | `default` | Runs unpinned tests once per selected agent. For a test with `{ agent }`, selects the shared names when there is overlap; otherwise keeps the test's selection. An unknown name is `INVALID_CONFIG`. |
@@ -366,7 +372,7 @@ or every test was filtered out.
 | `--repeat-each <n>` | integer, at least 1 | `1` | Runs every selected test this many times. Each run is a result of its own, with `repeat` 0 through `n - 1` and its own `id`; the list reporter titles the later runs `(repeat #1)`, `(repeat #2)`, and their artifacts go under a `repeat-<n>` directory. Setup tests run once. With the replay cache on, the first run records and the later ones replay, so pair it with `--no-cache` to exercise the model each time. The `list` summary adds a `Repeats` row (`1 of 2 tests passed all 5 runs`) and one line per test that did not pass every run, naming each run that failed; the `markdown` page carries the same tally. |
 | `--max-failures <n>` | integer, at least 1 | none | Stops the run once this many tests have failed or timed out. Nothing more is dispatched, the tests running on other workers end as `interrupted`, and the tests not started, the rest of the file whose test reached the limit included, are skipped with cause `failure-limit` and the reason `run stopped after 3 failures (--max-failures 3)`. The exit code is the failures' own, 1, not 130. |
 | `--reporter <ids>` | comma-separated list | config | Replaces the built-in IDs in `config.reporters`: `list`, `json`, `junit`, `markdown`. `json` with `list` is `INVALID_CONFIG`. [Reporter objects](/reference/reporters) in the config keep running. |
-| `--output <dir>` | string | config, else `.e2e` | The results directory for this run, over the config's [`output`](/reference/config#output): the report, the reporter files, the AI trace, `artifacts/` (cleared once the run's tests start), and `sessions/`. Resolved from the project root; one the run cannot own is `INVALID_CONFIG`. The removed `--artifacts <dir>` fails with the `--output` it maps to: `--artifacts out/artifacts` is `--output out`, and a directory whose parent is the project root or outside it maps to `--output .e2e`. |
+| `--output <dir>` | string | config, else `.e2e` | The results directory for this run, over the config's [`output`](/reference/config#output): the report, the reporter files, the AI trace, `artifacts/` (cleared once the run's tests start, down to what the previous report names on a `--last-failed` rerun), and `sessions/`. Resolved from the project root; one the run cannot own is `INVALID_CONFIG`. The removed `--artifacts <dir>` fails with the `--output` it maps to: `--artifacts out/artifacts` is `--output out`, and a directory whose parent is the project root or outside it maps to `--output .e2e`. |
 | `--no-cache` | boolean | `false` | Replay cache off for this run, whatever `config.cache` says. |
 | `--strict-cache` | boolean | `false` | Turns on `cache.strict` for this run: a step whose recording exists but no longer replays fails with `REPLAY_STALE` instead of handing off to the agent. See [Fail on a stale recording](/cache#fail-on-a-stale-recording). |
 | `--pass-with-no-tests` | boolean | `false` | Exit 0 instead of `NO_TESTS` when nothing is selected. |
@@ -477,8 +483,8 @@ bunx e2e run --reporter list,junit
 | --- | --- |
 | `list` | Terminal progress, failures with code frames, and a run summary. On a TTY, live progress includes the current agent step, model turns, tool calls, and reasoning excerpts. Untrusted fields are capped at 8 KiB and stripped of control characters. |
 | `json` | The report document to stdout and nothing else there, so `e2e run --reporter json \| jq` works. Same document as `<output>/report.json`, described by `schema/report-v1.schema.json` in the package. The shape changes only with the `schemaVersion` field (`report-1` today). Cannot be combined with `list`. |
-| `junit` | `junit.xml` beside the report. One `<testsuite>` per file, one `<testcase>` per test-target pair named `title > path [target]`. A test-class error is a `<failure>`, any other an `<error>`, a skip a `<skipped>` with its reason. Run-level errors form a `run` suite. |
-| `markdown` | `summary.md` beside the report with counts, cost, errors, and failure details. Failed and flaky tests also get a page under `failures/` there, with steps, model turns, and the screen at failure. Exploration runs show findings. [`@e2e-dev/github`](/github) uses this summary for pull request comments. |
+| `junit` | `junit.xml` beside the report. One `<testsuite>` per file, one `<testcase>` per test-target pair named `title > path [target]`. A test-class error is a `<failure>`, any other an `<error>`, a skip a `<skipped>` with its reason. An interrupted test reached no verdict, so it is a `<skipped>` whose message starts `interrupted:`. Run-level errors form a `run` suite. |
+| `markdown` | `summary.md` beside the report with counts, cost, errors, and failure details. Failed and flaky tests also get a page under `failures/` there, with steps, model turns, and the screen at failure. Interrupted tests are counted on their own and get no page. Exploration runs show findings. [`@e2e-dev/github`](/github) uses this summary for pull request comments. |
 
 Every run that gets to its tests writes `report.json` under the [output directory](/reference/config#output)
 (`.e2e` by default) whatever the reporters are, and the reporters' files
@@ -488,6 +494,16 @@ fails to start, an interrupt) writes none of them and leaves the previous
 run's files in place; its errors are on the terminal and in the exit code. A `junit.xml` or `summary.md` that could not be written is a
 line on stderr, never a run error.
 
+`run.summary` in the report counts the results. `discovered` is every
+test-target pair the runner collected and `selected` the ones this run
+chose; the selection's left-out pairs are in `results` as `selected: false`
+but in no other count. `passed`, `failed` (timed out included),
+`interrupted`, `flaky`, and `skipped` count the selected results and add up
+to `selected`. `interrupted` is a test the run stopped before its verdict
+(a signal, `--max-failures`, or a run-level error such as
+`MODEL_UNAVAILABLE`), never a failure. `executed` is the selected
+results that ran at least one attempt.
+
 With `--ai-trace`, the run also writes `<output>/ai-trace.json`: one entry per
 model round trip with the exact prompt, tool definitions, response, usage,
 and provider metadata. Observations in it are already redacted. Inline image
@@ -508,7 +524,10 @@ Signals escalate.
    any test starts (during collection, an engine's `prepare`, or app
    startup) writes no report and leaves the previous one in place.
 2. A second signal forces. Workers dispose their engines at once and are
-   killed when the cleanup budget is spent. The report is still written.
+   killed when the cleanup budget is spent. The report is still written,
+   and so are `junit.xml` and `summary.md` when the `junit` and
+   `markdown` reporters are on; a custom reporter's
+   `onRunFinished` is abandoned.
 3. A third signal exits on the spot, after killing the process groups of every
    app command the run started. A process attached through
    `reuseExisting` is left alone.

package/docs/reference/config.mdx

@@ -96,6 +96,8 @@ export interface E2EConfig {
   cleanupTimeout?: number;
   /** Retries per test, 0 through 10; default 1 in CI, else 0. */
   retries?: number;
+  /** Fail the run when a test skips itself after a soft failure or an earlier failed attempt; default false. */
+  failOnSkippedFailure?: boolean;
   /** Parallel workers, 1 through 1024; default 1 in CI, else half the cores. An engine may cap it lower. */
   workers?: number;
   /** `{ store }` hands every artifact to a host store as it is produced. */
@@ -158,6 +160,17 @@ Every numeric value must be a safe integer. `projectId` falls back to
 `package.json`. `workers` never exceeds what a target's engine declares. A
 targeted agent action is bounded by the smaller of `actionTimeout` and 15 s.
 
+## failOnSkippedFailure
+
+Set `failOnSkippedFailure: true` to fail the run with exit code 1 when a test
+skips after a soft assertion failure or skips a retry after an earlier failed
+attempt. The default is `false`.
+
+The test keeps its `skipped` status and reason. Its failures remain in the
+report, and the CLI shows them under `Skipped After Failure` with either
+setting. Clean skips, successful retries, and engine cleanup diagnostics do
+not trigger this policy. A skip still stops the body and does not retry.
+
 ## targets
 
 ```ts
@@ -267,8 +280,9 @@ it (no `app.command`) is `INVALID_CONFIG`. An `app.command` with neither
 
 ### The app's site
 
-A test or the agent may open any http(s) URL; `file:`, `data:`, and
-`javascript:` are `POLICY_DENIED`. The site of `url`, its registrable domain
+A test or the agent may open any http(s) URL, and `about:blank`; every
+other scheme (`file:`, `data:`, `javascript:`, `view-source:`, `blob:`) is
+`POLICY_DENIED`. The site of `url`, its registrable domain
 (`app.example.com` and `auth.example.com` are one site), scopes two things:
 the browser engine's [`headers`](/reference/web#options) reach the
 site and no other host, and child frames off the site stay out of
@@ -369,7 +383,7 @@ run.
 | `report.json` | The canonical report; `--last-failed` reads the one here |
 | `junit.xml`, `summary.md`, `failures/` | The `junit` and `markdown` reporters' files |
 | `ai-trace.json` | `--ai-trace` |
-| `artifacts/` | Screenshots, traces, videos, downloads, and logs, one directory per attempt. Cleared once a run's tests start, so it holds the latest run's evidence and nothing its report no longer names |
+| `artifacts/` | Screenshots, traces, videos, downloads, and logs, one directory per attempt. Cleared once a run's tests start, so it holds the latest run's evidence and nothing its report no longer names. A `--last-failed` rerun keeps the files the report it reruns names and writes its own attempts under `artifacts/rerun-<n>/` |
 | `sessions/` | The run's encrypted `test.setup` sessions, deleted when it ends |
 | `videos/<session>/` | [`e2e mcp`](/reference/mcp#recording) session recordings |
 
@@ -754,8 +768,9 @@ export interface ModelInstance {
 }
 ```
 
-Construct the model in your config with an AI SDK provider. For subscription
-sign-in, use the [`e2e/oauth/*` models](/subscriptions).
+Construct the model in your config with an AI SDK provider. You can use all
+providers that support tool calls, see [AI SDK providers](/models#ai-sdk-providers).
+For subscription sign-in, use the [`e2e/oauth/*` models](/subscriptions).
 
 | Gateway | Constructor | Package | Reads |
 | --- | --- | --- | --- |
@@ -798,7 +813,9 @@ Credentials and secrets are separate namespaces: `secrets.get(name)` returns a
 reports know that handle as `<name>.password` (`<secret:admin.password>`), so
 `credentials.admin` and `secrets.admin` can both exist; a `secrets` entry
 named exactly `admin.password` beside an `admin` credential is
-`INVALID_CONFIG`.
+`INVALID_CONFIG`. So are two credentials whose names map to one
+`E2E_USER_<NAME>_*` override, like `svc-a` and `svc_a` (see
+[Environment variables](/reference/environment)).
 
 ```ts
 /** Named test identity. The password remains an opaque Secret. */
@@ -824,7 +841,8 @@ export type SecretConfig = string | SecretProvider;
 
 An entry is a static value, or a provider called on every authorized fill (a
 vault lookup, a freshly computed TOTP). It is recorded in the config digest by
-name only, and an `E2E_SECRET_<NAME>` override wins over both. Every entry
+name only, and an `E2E_SECRET_<NAME>` override wins over both. Two entries whose
+names map to one variable, like `api-key` and `api_key`, are `INVALID_CONFIG`. Every entry
 must resolve to a string of at least 6 characters (code points) or a
 function; anything else is `INVALID_CONFIG`. The minimum exists for redaction: every occurrence
 of a value becomes `<secret:name>`, and a shorter value would take ordinary

package/docs/reference/engine.mdx

@@ -960,10 +960,11 @@ Use these helpers to keep behavior consistent across engines:
 | --- | --- |
 | `matchesText(actual, pattern)`, `toTextPattern(match, { exact? })`, `describePattern(pattern)` | `TextPattern` matching for `locate` and text matchers; `toTextPattern` throws `INVALID_ARGUMENT` for a `match` that is neither a string nor a `RegExp` |
 | `resolveExpression(expression, nodes, { selector? })` | The reference `locate`. Resolves one `LocatorExpression` over a semantic tree in document order, with `text` and `label` answering the innermost match when a container echoes a descendant's text. An engine whose tree is the whole truth (a device) calls it; a browser engine reproduces it. A `selector` needs the hook, a `frame` is `FRAME_NOT_FOUND` |
-| `urlMatches(current, expected, baseHref)` | `toHaveURL`-style comparison; an `expected` that is neither a string nor a `RegExp` throws `INVALID_ARGUMENT`, as in `toTextPattern` |
+| `urlMatches(current, expected, baseHref, ignoreCase?)` | `toHaveURL`-style comparison, `ignoreCase` as on the locator matchers; an `expected` that is neither a string nor a `RegExp` throws `INVALID_ARGUMENT`, as in `toTextPattern` |
 | `pollCondition({ deadline, signal, negated, evaluate, onTimeout })`, `Deadline` | Assertion polling and the negation grace window |
 | `validateJsonValue(value, label, rules?)` | Checks a value is plain JSON: a top-level `undefined` passes; a non-finite number, a cycle, an object that is not a plain object or array, or a function, symbol, bigint, or `undefined` inside is `INVALID_ARGUMENT` naming `label`. A `Secret` is `POLICY_DENIED` unless `rules.onSecret` throws its own; `rules.maxDepth` caps the nesting |
 | `rejectUnknownKeys(label, value, keys, code?)` | Throws `INVALID_CONFIG` for the first own key of `value` outside `keys`, naming the nearest known key when one is a plausible typo (`web() has unknown key "viewprt"; did you mean "viewport"?`) and the known keys otherwise: the check the runner runs on its own config objects, for an engine factory's options. Pass `'INVALID_ARGUMENT'` as `code` for an object a test passes to a fixture method |
+| `rejectUnknownOptions(api, options, known, code?)` | `INVALID_ARGUMENT` (or `code`, `INVALID_LOCATOR` for a locator option) for an option key a fixture method or matcher does not take, so a JavaScript caller's unsupported option fails instead of running on the default. `options` must be a plain object or `undefined`; a class instance or array is refused, and non-enumerable own keys count as keys |
 | `obj`, `raceAbort`, `withTimeout`, `withinCleanupBudget` | See [Contributed fixtures](#contributed-fixtures) |
 | `isProviderRecording(value)` | Whether what a provider's `record` resolved to is a recording: a parseable `startedAt` and a `stop`. Check it before keeping one, so a malformed recording fails the start, named, instead of the stop |
 | `stopProviderRecording(recording, { artifactsDir, provider, leaseId, signal })` | Ends a hosted service's recording and checks it into a `VideoSegment`; see [Provider recordings](#provider-recordings) |
@@ -1184,8 +1185,8 @@ export interface EngineFixtureContext {
     /**
      * Resolves a navigation target against the base URL. Throws
      * `APP_URL_REQUIRED` when the target declares no URL and `POLICY_DENIED`
-     * for a `file:`, `data:`, or `javascript:` scheme, so a fixture never
-     * re-implements the rule the harness owns.
+     * for any scheme but `http:` and `https:` (`about:blank` aside), so a
+     * fixture never re-implements the rule the harness owns.
      */
     resolveUrl(url: string): string;
   };

package/docs/reference/environment.mdx

@@ -21,6 +21,7 @@ See [Models](/models) and [Subscriptions](/subscriptions).
 | Variable | Reading | Effect |
 | --- | --- | --- |
 | `E2E_OAUTH_CREDENTIALS` | value | The stored subscription logins as JSON, the contents of `oauth.json`, for a machine nobody can sign in on. The `e2e/oauth/*` models and `e2e models` read it in place of `$XDG_CONFIG_HOME/e2e/oauth.json`; `e2e login` and `e2e logout` refuse to change it, and refreshed tokens are not written back |
+| `OPENCODE_API_KEY` | value | An OpenCode Console service account key; `opencodeConsole()` uses it in place of the stored `opencode-console` login; `e2e models` still reads the stored login. OpenCode reads the same variable |
 | `XDG_CONFIG_HOME` | value | Directory of the stored logins, `$XDG_CONFIG_HOME/e2e/oauth.json`; default `~/.config/e2e/oauth.json` |
 
 ## Credentials and secrets
@@ -32,7 +33,13 @@ See [Models](/models) and [Subscriptions](/subscriptions).
 | `E2E_SECRET_<NAME>` | set | Replaces the value of `secrets.<name>` |
 
 `<NAME>` is the name uppercased with every character outside `A-Z0-9`
-replaced by `_`, so `power-user` becomes `E2E_USER_POWER_USER_USERNAME`. A
+replaced by `_`, so `power-user` becomes `E2E_USER_POWER_USER_USERNAME`. Two
+credentials, or two secrets, whose names map to the same `<NAME>` (`api-key`
+and `api_key` both read `E2E_SECRET_API_KEY`; `svc-a` and `svc_a` both read
+`E2E_USER_SVC_A_USERNAME` and `_PASSWORD`) are `INVALID_CONFIG` at config
+load, whether or not a variable is set, so one rotation never replaces two
+entries; the message names the entries and the variables. A credential and a
+secret may share a name, since their prefixes differ. A
 variable for a name absent from the config is ignored. An empty value is
 `INVALID_CONFIG`, and so is a password or secret shorter than 6 characters
 (code points): redaction rewrites every occurrence of a value, and one that

package/docs/reference/errors.mdx

@@ -56,7 +56,7 @@ try {
 | `TEST_SETUP_FAILED` | Test prerequisites outside the product failed to arrange (blocked: test_setup). Also a `test.extend()` fixture that broke its contract: returned without calling `use()`, called it twice, or redefined a fixture the target's engine contributes | configuration | 2 |
 | `AUTOMATION_UNSUPPORTED` | The step needs an interaction the available toolset cannot perform, e.g. drag-and-drop (blocked: automation) | test | 1 |
 | `MODEL_UNAVAILABLE` | No model configured: the built-in agent's `agents.<name>.model` holds no AI SDK model instance; the message names the agent that was selected. Checked once per run when the `agent` fixture is first acquired; recorded as one run-level error that stops the run. `e2e explore` checks before anything starts, a custom executor's entry included, since explore runs the built-in agent. A custom executor without a model is not checked: its `act` and `assert` steps run, and `waitFor` or `extract` fails with this code on its first call. Evicts no [recording](/cache). | configuration | 2 |
-| `POLICY_DENIED` | A denied destination (`file:`, `data:`, `javascript:`, a malformed URL), a forbidden fill (an undeclared secret, a non-editable sink, a password into a field that is not a password field), or pixel evidence after a secret fill (`vision: 'only'` with pixels withheld). Also reading a value, text, or attribute from a secure field, the getters and the `toHaveValue`, `toHaveText`, `toContainText`, and `toHaveAttribute` matchers alike. An unknown tool or an undeclared field is refused to the model as the call's result and is not this code | configuration | 2 |
+| `POLICY_DENIED` | A denied destination (a navigation scheme other than `http:` or `https:`, `about:blank` aside, a malformed URL), a forbidden fill (an undeclared secret, a non-editable sink, a password into a field that is not a password field), or pixel evidence after a secret fill (`vision: 'only'` with pixels withheld). Also reading a value, text, or attribute from a secure field, the getters and the `toHaveValue`, `toHaveText`, `toContainText`, and `toHaveAttribute` matchers alike. An unknown tool or an undeclared field is refused to the model as the call's result and is not this code | configuration | 2 |
 | `APP_UNREACHABLE` | The app URL could not be reached, or an app command exited early or exceeded `startupTimeout`. The startup error includes recent log output with `command.env` values redacted, and suggests adding a `log` if none is configured. Under the agent, also any engine fault with no more specific code: on a device, an iOS automation runner that is busy or wedged reports here with a message that names the runner, the session and device, and the recovery; see [Testing iOS and Android](/mobile#troubleshooting) | infrastructure | 3 |
 | `APP_ALREADY_RUNNING` | `readyUrl` already answered before the target's `app.command` was spawned and `reuseExisting` was not in effect; stop that process or set `reuseExisting: true` locally | infrastructure | 3 |
 | `MODEL_PROVIDER_FAILED` | Provider transport failure (network error, 5xx, rate limit, or a request with no response within 120 s) after the adapter's transport retries (up to 5 per model call, honoring `retry-after`, bounded by the remaining step timeout). A request that stalls is aborted at 120 s and sent again, and an `agent.act` step's `--debug` transcript notes it. The bound covers the whole request, so a model that needs longer than 120 s for one answer fails here too. Evicts no [recording](/cache) | infrastructure | 3 |
@@ -73,7 +73,7 @@ try {
 | `ASSERTION_INCONCLUSIVE` | `agent.assert` could not judge: the screen did not show enough to decide either way (the relevant part is not on screen, still loading, or unreadable). `agent.extract` raises it when the model answers that the screen does not show the requested data, instead of returning a placeholder. A failure, never a pass; the explanation says what was missing. Judged from the tree alone with `vision` off, the message adds `pass vision: true when the answer is in pixels`, as a condition (`if the engine captures pixels, ...`) until a step of the attempt has received pixels; no hint when the engine declares no screenshot capture, a secret was filled, or an earlier step's pixel request was degraded | test | 1 |
 | `AUTHENTICATION_FAILED` | The executor concluded that signing in failed inside the app: a product verdict. `AUTH_CREDENTIAL_INVALID` is the blocked counterpart for a configured credential the app rejects | test | 1 |
 | `STEP_NO_CONCLUSION` | The executor used every turn without calling `complete_step`; the step fails with the agent's last summary | test | 1 |
-| `REPLAY_STALE` | Under `cache.strict` or `--strict-cache`, the step's recording exists but no longer replays (`step.cache.reason` is one of `invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `viewport-changed`, `action-failed`, `action-uncertain`, `end-mismatch`), so the step fails with no model call instead of handing off to the agent. Not retried: a retry never replays and would pass live. Re-record with a read-write run without the flag, and with `cache.strict` off when the config set it, then commit the changed entry; the message names the knob that turned strict on and the configured cache directory (or `cache.store`) | configuration | 2 |
+| `REPLAY_STALE` | Under `cache.strict` or `--strict-cache`, the step's recording exists but no longer replays (`step.cache.reason` is one of `invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `viewport-changed`, `action-failed`, `action-uncertain`, `end-mismatch`, or `no-entry` while the cache directory, not a custom `cache.store`, holds a recording of the step under another key, which the message names), so the step fails with no model call instead of handing off to the agent. Not retried: a retry never replays and would pass live. Re-record with a read-write run without the flag, and with `cache.strict` off when the config set it, then commit the changed entry; the message names the knob that turned strict on and the configured cache directory (or `cache.store`) | configuration | 2 |
 
 ## Runner codes
 
@@ -84,7 +84,7 @@ These appear in terminal output and in the report as plain string codes.
 | `CONFIG_NOT_FOUND` | `--config <path>` names a file that does not exist, or no `e2e.config.ts` or `e2e.config.mts` exists from the working directory up to the repository root | configuration | 2 |
 | `CONFIG_AMBIGUOUS` | Both `e2e.config.ts` and `e2e.config.mts` in one directory | configuration | 2 |
 | `CONFIG_LOAD_FAILED` | The config module failed to import: a package that is declared but not installed, an undeclared package, a wrong subpath, a removed export, or any other throw while it evaluates. The message names the fix. A config error raised while it evaluates keeps its own code: an option a factory the config calls (`web()`, `mobile()`, `defineEngine()`) refuses, or a `secrets.get()` handle turned into a string, is `INVALID_CONFIG` | configuration | 2 |
-| `INVALID_CONFIG` | A factory the config module calls (`web()`, `mobile()`, `defineEngine()`) refused its options, with the factory's own message; or an unknown key, out-of-range value, invalid enum, missing `targets`, a target key outside `{ name, platform, engine, app, trace, video }`, a target `app` key that is unknown or malformed, `app` on a target without an engine, a `web()` target without `app.url` or with a device field, a `mobile()` target without `app.bundleId` or `app.appPath`, with `app.url` (not supported yet), a link in `bundleId`, or an unknown permission name or state (the message names the nearest name), an unknown key in a `web()` or `mobile()` option, including `web({ connect })` and `web({ basicAuth })` (the message names the nearest one), a `trace` or `video` that is not a recording mode, the removed artifact kinds list (`artifacts: [...]` or `artifacts.kinds`) and `artifacts.trace` block (the message names the `trace` mode they meant), the removed `artifacts.video` block or `video` artifact kind (the message names the `video` option), the removed `specVersion` key, a `secrets.get()` in an engine option that names no `secrets` entry (a credential's password is not one), a `secrets.get()` reference turned into a string while the config evaluates (a template literal, `String()`) or given where a string is needed (an `app.command` `args` entry or `env` value; only an engine option that declares secrets takes one), an `app.command` `args` entry or `env` value that is not a string (an `undefined` `env` value is dropped instead), an unknown key in `app.command` (the message names the nearest one), an `engine` that is not a `defineEngine` handle, `json` reporter combined with `list`, duplicate or malformed target name, a secret or credential password shorter than 6 characters (code points), oversized `agents.<name>.context`, an `expect.poll` `timeout` or `interval` that is not a finite number of milliseconds, a `tests` value that is not a glob or a list of globs, a `tests` list of only `!` exclusions, the removed top-level `limits` key, an agents entry that is a bare `StepExecutor` (the message names `{ executor }`) or a model instance (the message names `{ model }`), an unknown agent key (the message names the nearest one), the removed agent keys `timeout` and `maxTurns` (the message names `judgmentTimeout` and `maxModelCalls`), `system` or `tools` beside `executor`, a `tools` entry `defineTool` did not build or that takes a built-in tool name (a grammar verb, `complete_step`, `locate`, `start_recording`, `stop_recording`, or `report_finding`), `createToolLoopExecutor({ maxTurns })`, `reuseExisting` on an app whose `app.url` asks for a free port, or a port-0 `app.url` with no `app.command` to serve it | configuration | 2 |
+| `INVALID_CONFIG` | A factory the config module calls (`web()`, `mobile()`, `defineEngine()`) refused its options, with the factory's own message; or an unknown key, out-of-range value, invalid enum, missing `targets`, a target key outside `{ name, platform, engine, app, trace, video }`, a target `app` key that is unknown or malformed, `app` on a target without an engine, a `web()` target without `app.url` or with a device field, a `mobile()` target without `app.bundleId` or `app.appPath`, with `app.url` (not supported yet), a link in `bundleId`, or an unknown permission name or state (the message names the nearest name), an unknown key in a `web()` or `mobile()` option, including `web({ connect })` and `web({ basicAuth })` (the message names the nearest one), a `trace` or `video` that is not a recording mode, the removed artifact kinds list (`artifacts: [...]` or `artifacts.kinds`) and `artifacts.trace` block (the message names the `trace` mode they meant), the removed `artifacts.video` block or `video` artifact kind (the message names the `video` option), the removed `specVersion` key, a `secrets.get()` in an engine option that names no `secrets` entry (a credential's password is not one), a `secrets.get()` reference turned into a string while the config evaluates (a template literal, `String()`) or given where a string is needed (an `app.command` `args` entry or `env` value; only an engine option that declares secrets takes one), an `app.command` `args` entry or `env` value that is not a string (an `undefined` `env` value is dropped instead), an unknown key in `app.command` (the message names the nearest one), an `engine` that is not a `defineEngine` handle, `json` reporter combined with `list`, duplicate or malformed target name, a secret or credential password shorter than 6 characters (code points), two credentials or two secrets whose names map to one `E2E_USER_*` or `E2E_SECRET_*` override variable (`api-key` and `api_key`), oversized `agents.<name>.context`, an `expect.poll` `timeout` or `interval` that is not a finite number of milliseconds, a `tests` value that is not a glob or a list of globs, a `tests` list of only `!` exclusions, the removed top-level `limits` key, an agents entry that is a bare `StepExecutor` (the message names `{ executor }`) or a model instance (the message names `{ model }`), an unknown agent key (the message names the nearest one), the removed agent keys `timeout` and `maxTurns` (the message names `judgmentTimeout` and `maxModelCalls`), `system` or `tools` beside `executor`, a `tools` entry `defineTool` did not build or that takes a built-in tool name (a grammar verb, `complete_step`, `locate`, `start_recording`, `stop_recording`, or `report_finding`), `createToolLoopExecutor({ maxTurns })`, `reuseExisting` on an app whose `app.url` asks for a free port, or a port-0 `app.url` with no `app.command` to serve it | configuration | 2 |
 | `CONFIG_NOT_DETERMINISTIC` | A worker re-resolved the config module and got a different digest than the runner | configuration | 2 |
 | `APP_URL_REQUIRED` | A navigation or relative URL on a target that declares no `app.url`, or an `app.command` with no `app.url` or `app.readyUrl` to poll | configuration | 2 |
 | `INVALID_APP_URL` | Base URL with userinfo, a query, a fragment, a non-HTTP scheme, plain HTTP on a non-loopback host, port 0 on a host other than a literal loopback address (`127.0.0.1`, `[::1]`), a loopback address the run could not bind a free port on, or unparseable | configuration | 2 |
@@ -104,14 +104,14 @@ These appear in terminal output and in the report as plain string codes.
 | `NO_LAST_RUN` | `--last-failed` found no `report.json` where the run writes it, or a file there that is not a `report-1` document. Run once without the flag first. | configuration | 2 |
 | `ONLY_IN_CI` | A focused test reached a CI run | configuration | 2 |
 | `COLLECTION_ERROR` | Registration outside collection, an invalid title, a non-function body, an async `describe` body, nested serial groups, an option forbidden inside a serial group, a setup test inside `describe`, an invalid session name, a malformed test option such as `tags` that is not a list of distinct names, a test or `describe` option key the call does not take (the message names the nearest one), a test file that fails to import (in a run with no positionals, or a file the positionals selected), a session that a selected test consumes or `e2e explore --session` names but no collected setup test produces (the message lists the declared sessions), a consumed session on a target whose engine has no `state` capability to save it, or a positional path outside the project root | configuration | 2 |
-| `POLICY_DENIED` | A denied destination (`file:`, `data:`, `javascript:`, a malformed URL), a forbidden fill (an undeclared secret, a non-editable sink, a mismatched purpose, a `Secret` reaching a JSON sink), or tainted pixels (`app.screenshot()` after a secret fill, `vision: 'only'` with pixels withheld); see the [security model](/security). An unknown tool or an undeclared field goes back to the model as the call's result, never this code | configuration | 2 |
+| `POLICY_DENIED` | A denied destination (a navigation scheme other than `http:` or `https:`, `about:blank` aside, a malformed URL), a forbidden fill (an undeclared secret, a non-editable sink, a mismatched purpose, a `Secret` reaching a JSON sink), or tainted pixels (`app.screenshot()` after a secret fill, `vision: 'only'` with pixels withheld); see the [security model](/security). An unknown tool or an undeclared field goes back to the model as the call's result, never this code | configuration | 2 |
 | `SESSION_MISMATCH` | A stored session does not match the current run, target, engine, platform, or app identity | configuration | 2 |
 | `SESSION_EXPIRED` | A stored session is past its expiry at restore time (configuration, exit 2), or a setup test saved state whose expiry is not after its creation (test, exit 1) | configuration or test | 2 or 1 |
 | `SESSION_INVALID` | A stored session failed authentication. Tampered or corrupt. | configuration | 2 |
 | `SESSION_UNAVAILABLE` | A test requires a session no selected setup test produced | configuration | 2 |
 | `SESSION_CONTRACT` | A setup test saved a session twice, saved an undeclared name, or finished without saving a declared one | test | 1 |
 | `TEST_TIMEOUT` | The attempt exceeded the resolved test `timeout`. The body is not stopped: the harness operation it is in is cancelled and its fixtures die with the attempt, so it cannot reach the next test | test | 1 |
-| `STEP_NOT_AWAITED` | The test body settled while a step it started was still running: a step call without `await`. The step is failed at the line of the call and cancelled; the attempt fails with this code when the body returned, or keeps the body's own error and records this one under `secondaryErrors` when the body threw | test | 1 |
+| `STEP_NOT_AWAITED` | The test body settled while a step it started was still running: a step call without `await`. The step is failed at the line of the call and cancelled; the attempt fails with this code when the body returned, or keeps the body's own error and records this one under `secondaryErrors` when the body threw. An `expect.poll` matcher left running when the body, an `afterEach` hook, or a fixture teardown returned fails that phase the same way at the line of the call, and is cancelled; one left by a `beforeAll` or `afterAll` hook fails the hook with `HOOK_FAILED` | test | 1 |
 | `UNHANDLED_REJECTION` | A promise rejected with no test running. An unhandled rejection is charged to the test running when it surfaces, since Node cannot say which test created the promise; between tests it is this run error, naming the last test that finished in the worker as the likely source. Before any test has finished in the worker it ends the worker instead: the rejection is the run error and the test on its way fails with `WORKER_CRASH`. The body that left it keeps running like one past `TEST_TIMEOUT` | test | 1 |
 | `HOOK_FAILED` | `beforeAll` or `afterAll` threw. Skips the suite's tests. | test | 1 |
 | `LAUNCH_TIMEOUT` | Engine init or attempt start exceeded `config.launchTimeout` | infrastructure | 3 |