e2e 0.16.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -0
- package/dist/.build.json +1 -1
- package/dist/agent/act.js +7 -4
- package/dist/agent/action-dispatcher.js +4 -5
- package/dist/agent/actions.d.ts +32 -23
- package/dist/agent/actions.js +38 -24
- package/dist/agent/ai-sdk.js +5 -4
- package/dist/agent/error.js +3 -3
- package/dist/agent/invocation.js +3 -4
- package/dist/agent/model/sdk.js +2 -0
- package/dist/agent/model/tool-choice.d.ts +9 -0
- package/dist/agent/model/tool-choice.js +14 -0
- package/dist/agent/observation-feed.d.ts +5 -6
- package/dist/agent/observation-feed.js +8 -11
- package/dist/agent/observation.d.ts +40 -11
- package/dist/agent/observation.js +109 -59
- package/dist/agent/phases.d.ts +7 -1
- package/dist/agent/phases.js +12 -3
- package/dist/agent/point-tap.d.ts +8 -8
- package/dist/agent/point-tap.js +1 -1
- package/dist/agent/prompts.js +3 -0
- package/dist/agent/replay.d.ts +19 -8
- package/dist/agent/replay.js +29 -29
- package/dist/agent/scroll-target.d.ts +2 -3
- package/dist/agent/step-cache.d.ts +43 -15
- package/dist/agent/step-cache.js +156 -67
- package/dist/agent/tool-loop.js +94 -5
- package/dist/cache/anchors.d.ts +72 -23
- package/dist/cache/anchors.js +233 -60
- package/dist/cache/context.d.ts +25 -14
- package/dist/cache/context.js +50 -8
- package/dist/cache/decide.d.ts +2 -2
- package/dist/cache/decide.js +4 -6
- package/dist/cache/identity.d.ts +18 -4
- package/dist/cache/identity.js +9 -4
- package/dist/cache/recorder.d.ts +15 -1
- package/dist/cache/recorder.js +36 -23
- package/dist/cache/rekeyed.d.ts +46 -0
- package/dist/cache/rekeyed.js +89 -0
- package/dist/cache/relocate.d.ts +14 -14
- package/dist/cache/relocate.js +33 -25
- package/dist/cache/route.d.ts +32 -33
- package/dist/cache/route.js +94 -50
- package/dist/cache/store.d.ts +6 -0
- package/dist/cache/store.js +21 -1
- package/dist/cache/trace.d.ts +53 -14
- package/dist/cache/trace.js +76 -30
- package/dist/cli/bin.js +2 -2
- package/dist/cli/index.js +43 -32
- package/dist/cli/init/engines.d.ts +1 -3
- package/dist/cli/init/engines.js +2 -2
- package/dist/cli/init/gateways.d.ts +1 -1
- package/dist/cli/init/gateways.js +11 -1
- package/dist/cli/init/mcp-config.d.ts +2 -1
- package/dist/cli/init/mcp-config.js +8 -3
- package/dist/cli/init/sibling-versions.json +1 -1
- package/dist/cli/init/versions.d.ts +0 -8
- package/dist/cli/init/versions.js +1 -10
- package/dist/cli/mcp.d.ts +3 -2
- package/dist/cli/mcp.js +52 -16
- package/dist/cli/run-output.d.ts +17 -0
- package/dist/cli/run-output.js +45 -0
- package/dist/collect/collect.js +8 -4
- package/dist/collect/registry.js +27 -16
- package/dist/config/compiled-files.d.ts +36 -0
- package/dist/config/compiled-files.js +50 -0
- package/dist/config/diagnose.js +21 -19
- package/dist/config/esm-hooks.d.ts +39 -44
- package/dist/config/esm-hooks.js +238 -63
- package/dist/config/load.d.ts +1 -7
- package/dist/config/load.js +23 -66
- package/dist/config/register.d.ts +8 -0
- package/dist/config/register.js +9 -0
- package/dist/config/resolve.d.ts +1 -0
- package/dist/config/resolve.js +33 -1
- package/dist/config/specifiers.d.ts +19 -0
- package/dist/config/specifiers.js +33 -0
- package/dist/config/tsconfig.d.ts +56 -0
- package/dist/config/tsconfig.js +115 -0
- package/dist/config/typescript.d.ts +19 -0
- package/dist/config/typescript.js +253 -0
- package/dist/engine/contract.d.ts +1 -0
- package/dist/engine/index.d.ts +4 -2
- package/dist/engine/session.js +2 -2
- package/dist/expect/async.js +4 -2
- package/dist/expect/poll-scope.d.ts +52 -0
- package/dist/expect/poll-scope.js +113 -0
- package/dist/expect/poll.js +33 -6
- package/dist/expect/values.js +11 -4
- package/dist/internal/abandoned.d.ts +11 -0
- package/dist/internal/abandoned.js +23 -0
- package/dist/internal/brands.d.ts +2 -0
- package/dist/internal/brands.js +2 -0
- package/dist/internal/client-identity.d.ts +9 -0
- package/dist/internal/client-identity.js +14 -0
- package/dist/internal/errors.d.ts +10 -3
- package/dist/internal/errors.js +23 -8
- package/dist/internal/node-version.d.ts +16 -7
- package/dist/internal/node-version.js +50 -15
- package/dist/internal/package-manager.js +1 -1
- package/dist/internal/realm-slot.d.ts +5 -4
- package/dist/internal/redact.d.ts +9 -4
- package/dist/internal/redact.js +120 -23
- package/dist/internal/time.d.ts +26 -4
- package/dist/internal/time.js +58 -14
- package/dist/internal/urls.d.ts +9 -5
- package/dist/internal/urls.js +19 -10
- package/dist/locator/engine.d.ts +12 -2
- package/dist/locator/engine.js +55 -14
- package/dist/locator/screen.js +59 -9
- package/dist/mcp/catalog.d.ts +3 -2
- package/dist/mcp/catalog.js +6 -6
- package/dist/mcp/output.d.ts +59 -0
- package/dist/mcp/output.js +129 -0
- package/dist/mcp/server.d.ts +4 -0
- package/dist/mcp/server.js +9 -12
- package/dist/mcp/session.d.ts +8 -1
- package/dist/mcp/session.js +56 -15
- package/dist/mcp/tools.js +7 -5
- package/dist/oauth/chatgpt.js +3 -23
- package/dist/oauth/cli.js +1 -0
- package/dist/oauth/copilot.d.ts +5 -4
- package/dist/oauth/copilot.js +80 -7
- package/dist/oauth/grok.js +2 -1
- package/dist/oauth/models.js +2 -1
- package/dist/oauth/opencode-console.d.ts +10 -0
- package/dist/oauth/opencode-console.js +133 -0
- package/dist/oauth/providers/github-copilot.d.ts +12 -0
- package/dist/oauth/providers/github-copilot.js +77 -23
- package/dist/oauth/providers/opencode-console.d.ts +31 -0
- package/dist/oauth/providers/opencode-console.js +148 -0
- package/dist/oauth/providers.d.ts +2 -6
- package/dist/oauth/providers.js +3 -7
- package/dist/oauth/responses.d.ts +9 -0
- package/dist/oauth/responses.js +23 -0
- package/dist/report/build.d.ts +26 -0
- package/dist/report/build.js +16 -8
- package/dist/report/failure-text.d.ts +6 -1
- package/dist/report/failure-text.js +14 -4
- package/dist/report/format.d.ts +10 -5
- package/dist/report/format.js +12 -5
- package/dist/report/junit.js +11 -12
- package/dist/report/list.d.ts +12 -0
- package/dist/report/list.js +61 -24
- package/dist/report/markdown.d.ts +1 -1
- package/dist/report/markdown.js +37 -14
- package/dist/report/repeats.d.ts +6 -1
- package/dist/report/repeats.js +12 -3
- package/dist/run/artifacts.d.ts +22 -4
- package/dist/run/artifacts.js +71 -3
- package/dist/run/execute.d.ts +34 -5
- package/dist/run/execute.js +93 -44
- package/dist/run/extended-fixtures.js +6 -2
- package/dist/run/failure-evidence.js +10 -4
- package/dist/run/fixtures.js +6 -8
- package/dist/run/in-process.d.ts +2 -0
- package/dist/run/in-process.js +52 -10
- package/dist/run/last-run.d.ts +42 -8
- package/dist/run/last-run.js +121 -14
- package/dist/run/process-output.d.ts +73 -0
- package/dist/run/process-output.js +130 -0
- package/dist/run/provision.d.ts +2 -0
- package/dist/run/provision.js +1 -0
- package/dist/run/realm.d.ts +4 -1
- package/dist/run/realm.js +20 -6
- package/dist/run/records.d.ts +20 -4
- package/dist/run/records.js +26 -0
- package/dist/run/retry.d.ts +10 -1
- package/dist/run/retry.js +30 -12
- package/dist/run/runner.d.ts +9 -1
- package/dist/run/runner.js +101 -39
- package/dist/run/scheduler.js +255 -19
- package/dist/run/secrecy.d.ts +22 -19
- package/dist/run/secrecy.js +22 -18
- package/dist/run/serial.d.ts +75 -2
- package/dist/run/serial.js +243 -59
- package/dist/run/standalone.js +1 -1
- package/dist/run/steps.d.ts +0 -6
- package/dist/run/steps.js +3 -20
- package/dist/run/trace-redaction.d.ts +5 -6
- package/dist/run/trace-redaction.js +5 -6
- package/dist/run/worker/entry.js +40 -18
- package/dist/run/worker/handle.js +10 -5
- package/dist/run/worker/protocol.d.ts +85 -3
- package/dist/run/worker/session.d.ts +2 -0
- package/dist/run/worker/session.js +29 -0
- package/dist/telemetry/environment.d.ts +2 -1
- package/dist/telemetry/environment.js +12 -2
- package/dist/telemetry/events.js +1 -0
- package/dist/telemetry/project.d.ts +5 -6
- package/dist/telemetry/project.js +6 -11
- package/dist/telemetry/store.d.ts +5 -5
- package/dist/telemetry/store.js +5 -7
- package/dist/telemetry/telemetry.d.ts +2 -3
- package/dist/telemetry/telemetry.js +2 -4
- package/dist/types.d.ts +15 -6
- package/docs/agent-steps.mdx +1 -0
- package/docs/agents.mdx +7 -3
- package/docs/api-testing.mdx +2 -1
- package/docs/authentication.mdx +6 -3
- package/docs/browser.mdx +49 -12
- package/docs/bug-bash.mdx +2 -1
- package/docs/cache.mdx +66 -19
- package/docs/ci.mdx +14 -9
- package/docs/coding-agents.mdx +2 -1
- package/docs/core-concepts.mdx +208 -0
- package/docs/debugging.mdx +25 -12
- package/docs/decision-models.mdx +142 -0
- package/docs/example-projects.mdx +32 -0
- package/docs/executors.mdx +7 -2
- package/docs/explore.mdx +1 -0
- package/docs/github.mdx +14 -5
- package/docs/index.mdx +9 -5
- package/docs/integrations/eas.mdx +1 -0
- package/docs/integrations/index.mdx +1 -0
- package/docs/integrations/kernel.mdx +3 -2
- package/docs/migrate/cypress.mdx +26 -3
- package/docs/migrate/detox.mdx +24 -1
- package/docs/migrate/maestro.mdx +28 -4
- package/docs/migrate/playwright.mdx +44 -18
- package/docs/migrate/selenium.mdx +27 -3
- package/docs/mobile-ci.mdx +6 -5
- package/docs/mobile.mdx +96 -8
- package/docs/models.mdx +60 -9
- package/docs/quickstart.mdx +28 -16
- package/docs/reference/agent.mdx +13 -3
- package/docs/reference/app.mdx +2 -1
- package/docs/reference/cli.mdx +41 -18
- package/docs/reference/config.mdx +67 -13
- package/docs/reference/engine.mdx +25 -13
- package/docs/reference/environment.mdx +10 -2
- package/docs/reference/errors.mdx +13 -10
- package/docs/reference/expect.mdx +36 -13
- package/docs/reference/mcp.mdx +21 -8
- package/docs/reference/mobile.mdx +24 -13
- package/docs/reference/reporters.mdx +9 -2
- package/docs/reference/screen.mdx +49 -16
- package/docs/reference/test.mdx +32 -5
- package/docs/reference/web.mdx +133 -23
- package/docs/security.mdx +58 -42
- package/docs/subscriptions.mdx +92 -12
- package/docs/telemetry.mdx +9 -6
- package/docs/tools.mdx +1 -0
- package/docs/{starting-your-app.mdx → web.mdx} +157 -11
- package/docs/writing-an-engine.mdx +2 -1
- package/package.json +25 -8
- package/schema/report-v1.schema.json +14 -1
- package/skills/e2e/SKILL.md +5 -3
- package/skills/e2e/references/agent.md +33 -11
- package/skills/e2e/references/debugging.md +4 -4
- package/skills/e2e/references/mcp.md +8 -3
- package/skills/e2e/references/running.md +15 -9
- package/skills/e2e/references/setup.md +45 -16
- package/skills/e2e/references/writing-tests.md +35 -18
- package/docs/viewports.mdx +0 -146
- package/docs/writing-tests.mdx +0 -256
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Errors
|
|
3
3
|
description: Every code you can see, its class, and its exit code.
|
|
4
|
+
icon: "triangle-exclamation"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
Every error carries a class. The class determines the exit code; the code
|
|
@@ -56,8 +57,8 @@ try {
|
|
|
56
57
|
| `TEST_SETUP_FAILED` | Test prerequisites outside the product failed to arrange (blocked: test_setup). Also a `test.extend()` fixture that broke its contract: returned without calling `use()`, called it twice, or redefined a fixture the target's engine contributes | configuration | 2 |
|
|
57
58
|
| `AUTOMATION_UNSUPPORTED` | The step needs an interaction the available toolset cannot perform, e.g. drag-and-drop (blocked: automation) | test | 1 |
|
|
58
59
|
| `MODEL_UNAVAILABLE` | No model configured: the built-in agent's `agents.<name>.model` holds no AI SDK model instance; the message names the agent that was selected. Checked once per run when the `agent` fixture is first acquired; recorded as one run-level error that stops the run. `e2e explore` checks before anything starts, a custom executor's entry included, since explore runs the built-in agent. A custom executor without a model is not checked: its `act` and `assert` steps run, and `waitFor` or `extract` fails with this code on its first call. Evicts no [recording](/cache). | configuration | 2 |
|
|
59
|
-
| `POLICY_DENIED` | A denied destination (`
|
|
60
|
-
| `APP_UNREACHABLE` | The app URL could not be reached, or an app command exited early or exceeded `startupTimeout`. The startup error includes recent log output with `command.env` values redacted, and suggests adding a `log` if none is configured. Under the agent, also any engine fault with no more specific code: on a device, an iOS automation runner that is busy or wedged reports here with a message that names the runner, the session and device, and the recovery; see [
|
|
60
|
+
| `POLICY_DENIED` | A denied destination (a navigation scheme other than `http:` or `https:`, `about:blank` aside, a malformed URL), a forbidden fill (an undeclared secret, a non-editable sink, a password into a field that is not a password field), or pixel evidence after a secret fill (`vision: 'only'` with pixels withheld, or an `act` or `vision: true` step whose semantic capture timed out before the step did). Also reading a value, text, or attribute from a secure field, the getters and the `toHaveValue`, `toHaveText`, `toContainText`, and `toHaveAttribute` matchers alike. An unknown tool or an undeclared field is refused to the model as the call's result and is not this code | configuration | 2 |
|
|
61
|
+
| `APP_UNREACHABLE` | The app URL could not be reached, or an app command exited early or exceeded `startupTimeout`. The startup error includes recent log output with `command.env` values redacted, and suggests adding a `log` if none is configured. Under the agent, also any engine fault with no more specific code: on a device, an iOS automation runner that is busy or wedged reports here with a message that names the runner, the session and device, and the recovery; see [Mobile](/mobile#troubleshooting) | infrastructure | 3 |
|
|
61
62
|
| `APP_ALREADY_RUNNING` | `readyUrl` already answered before the target's `app.command` was spawned and `reuseExisting` was not in effect; stop that process or set `reuseExisting: true` locally | infrastructure | 3 |
|
|
62
63
|
| `MODEL_PROVIDER_FAILED` | Provider transport failure (network error, 5xx, rate limit, or a request with no response within 120 s) after the adapter's transport retries (up to 5 per model call, honoring `retry-after`, bounded by the remaining step timeout). A request that stalls is aborted at 120 s and sent again, and an `agent.act` step's `--debug` transcript notes it. The bound covers the whole request, so a model that needs longer than 120 s for one answer fails here too. Evicts no [recording](/cache) | infrastructure | 3 |
|
|
63
64
|
| `CANCELLED` | The runner cancelled the operation, without a process signal: an `e2e mcp` `open_session` whose request the client cancelled or whose connection closed, or that the server's shutdown cut short, after stopping what it had started | infrastructure | 3 |
|
|
@@ -73,7 +74,7 @@ try {
|
|
|
73
74
|
| `ASSERTION_INCONCLUSIVE` | `agent.assert` could not judge: the screen did not show enough to decide either way (the relevant part is not on screen, still loading, or unreadable). `agent.extract` raises it when the model answers that the screen does not show the requested data, instead of returning a placeholder. A failure, never a pass; the explanation says what was missing. Judged from the tree alone with `vision` off, the message adds `pass vision: true when the answer is in pixels`, as a condition (`if the engine captures pixels, ...`) until a step of the attempt has received pixels; no hint when the engine declares no screenshot capture, a secret was filled, or an earlier step's pixel request was degraded | test | 1 |
|
|
74
75
|
| `AUTHENTICATION_FAILED` | The executor concluded that signing in failed inside the app: a product verdict. `AUTH_CREDENTIAL_INVALID` is the blocked counterpart for a configured credential the app rejects | test | 1 |
|
|
75
76
|
| `STEP_NO_CONCLUSION` | The executor used every turn without calling `complete_step`; the step fails with the agent's last summary | test | 1 |
|
|
76
|
-
| `REPLAY_STALE` | Under `cache.strict` or `--strict-cache`, the step's recording exists but no longer replays (`step.cache.reason` is one of `invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `viewport-changed`, `action-failed`, `action-uncertain`, `end-mismatch`), so the step fails with no model call instead of handing off to the agent. Not retried: a retry never replays and would pass live. Re-record with a read-write run without the flag, and with `cache.strict` off when the config set it, then commit the changed entry; the message names the knob that turned strict on and the configured cache directory (or `cache.store`) | configuration | 2 |
|
|
77
|
+
| `REPLAY_STALE` | Under `cache.strict` or `--strict-cache`, the step's recording exists but no longer replays (`step.cache.reason` is one of `invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `viewport-changed`, `action-failed`, `action-uncertain`, `end-mismatch`, or `no-entry` while the cache directory, not a custom `cache.store`, holds a recording of the step under another key, which the message names), so the step fails with no model call instead of handing off to the agent. Not retried: a retry never replays and would pass live. Re-record with a read-write run without the flag, and with `cache.strict` off when the config set it, then commit the changed entry; the message names the knob that turned strict on and the configured cache directory (or `cache.store`) | configuration | 2 |
|
|
77
78
|
|
|
78
79
|
## Runner codes
|
|
79
80
|
|
|
@@ -83,8 +84,8 @@ These appear in terminal output and in the report as plain string codes.
|
|
|
83
84
|
| --- | --- | --- | ---: |
|
|
84
85
|
| `CONFIG_NOT_FOUND` | `--config <path>` names a file that does not exist, or no `e2e.config.ts` or `e2e.config.mts` exists from the working directory up to the repository root | configuration | 2 |
|
|
85
86
|
| `CONFIG_AMBIGUOUS` | Both `e2e.config.ts` and `e2e.config.mts` in one directory | configuration | 2 |
|
|
86
|
-
| `CONFIG_LOAD_FAILED` | The config module failed to import: a package that is declared but not installed, an undeclared package, a wrong subpath, a removed export, or any other throw while it evaluates. The message names the fix. A config error raised while it evaluates keeps its own code: an option a factory the config calls (`web()`, `mobile()`, `defineEngine()`) refuses, or a `secrets.get()` handle turned into a string, is `INVALID_CONFIG` | configuration | 2 |
|
|
87
|
-
| `INVALID_CONFIG` | A factory the config module calls (`web()`, `mobile()`, `defineEngine()`) refused its options, with the factory's own message; or an unknown key, out-of-range value, invalid enum, missing `targets`, a target key outside `{ name, platform, engine, app, trace, video }`, a target `app` key that is unknown or malformed, `app` on a target without an engine, a `web()` target without `app.url` or with a device field, a `mobile()` target without `app.bundleId` or `app.appPath`, with `app.url` (not supported yet), a link in `bundleId`, or an unknown permission name or state (the message names the nearest name), an unknown key in a `web()` or `mobile()` option, including `web({ connect })` and `web({ basicAuth })` (the message names the nearest one), a `trace` or `video` that is not a recording mode, the removed artifact kinds list (`artifacts: [...]` or `artifacts.kinds`) and `artifacts.trace` block (the message names the `trace` mode they meant), the removed `artifacts.video` block or `video` artifact kind (the message names the `video` option), the removed `specVersion` key, a `secrets.get()` in an engine option that names no `secrets` entry (a credential's password is not one), a `secrets.get()` reference turned into a string while the config evaluates (a template literal, `String()`) or given where a string is needed (an `app.command` `args` entry or `env` value; only an engine option that declares secrets takes one), an `app.command` `args` entry or `env` value that is not a string (an `undefined` `env` value is dropped instead), an unknown key in `app.command` (the message names the nearest one), an `engine` that is not a `defineEngine` handle, `json` reporter combined with `list`, duplicate or malformed target name, a secret or credential password shorter than 6 characters (code points), oversized `agents.<name>.context`, an `expect.poll` `timeout` or `interval` that is not a finite number of milliseconds, a `tests` value that is not a glob or a list of globs, a `tests` list of only `!` exclusions, the removed top-level `limits` key, an agents entry that is a bare `StepExecutor` (the message names `{ executor }`) or a model instance (the message names `{ model }`), an unknown agent key (the message names the nearest one), the removed agent keys `timeout` and `maxTurns` (the message names `judgmentTimeout` and `maxModelCalls`), `system` or `tools` beside `executor`, a `tools` entry `defineTool` did not build or that takes a built-in tool name (a grammar verb, `complete_step`, `locate`, `start_recording`, `stop_recording`, or `report_finding`), `createToolLoopExecutor({ maxTurns })`, `reuseExisting` on an app whose `app.url` asks for a free port, or a port-0 `app.url` with no `app.command` to serve it | configuration | 2 |
|
|
87
|
+
| `CONFIG_LOAD_FAILED` | The config module failed to import: a package that is declared but not installed, an undeclared package, a wrong subpath, a removed export, or any other throw while it evaluates. The message names the fix. A config error raised while it evaluates keeps its own code: an option a factory the config calls (`web()`, `mobile()`, `defineEngine()`, `decisionExecutor()`) refuses, or a `secrets.get()` handle turned into a string, is `INVALID_CONFIG` | configuration | 2 |
|
|
88
|
+
| `INVALID_CONFIG` | A factory the config module calls (`web()`, `mobile()`, `defineEngine()`, `decisionExecutor()`) refused its options, with the factory's own message; or an unknown key, out-of-range value, invalid enum, missing `targets`, a target key outside `{ name, platform, engine, app, trace, video }`, a target `app` key that is unknown or malformed, `app` on a target without an engine, a `web()` target without `app.url` or with a device field, a `mobile()` target without `app.bundleId` or `app.appPath`, with `app.url` (not supported yet), a link in `bundleId`, or an unknown permission name or state (the message names the nearest name), an unknown key in a `web()` or `mobile()` option, including `web({ connect })` and `web({ basicAuth })` (the message names the nearest one), a `trace` or `video` that is not a recording mode, the removed artifact kinds list (`artifacts: [...]` or `artifacts.kinds`) and `artifacts.trace` block (the message names the `trace` mode they meant), the removed `artifacts.video` block or `video` artifact kind (the message names the `video` option), the removed `specVersion` key, a `secrets.get()` in an engine option that names no `secrets` entry (a credential's password is not one), a `secrets.get()` reference turned into a string while the config evaluates (a template literal, `String()`) or given where a string is needed (an `app.command` `args` entry or `env` value; only an engine option that declares secrets takes one), an `app.command` `args` entry or `env` value that is not a string (an `undefined` `env` value is dropped instead), an unknown key in `app.command` (the message names the nearest one), an `engine` that is not a `defineEngine` handle, `json` reporter combined with `list`, duplicate or malformed target name, a secret or credential password shorter than 6 characters (code points), two credentials or two secrets whose names map to one `E2E_USER_*` or `E2E_SECRET_*` override variable (`api-key` and `api_key`), oversized `agents.<name>.context`, an `expect.poll` `timeout` or `interval` that is not a finite number of milliseconds, a `tests` value that is not a glob or a list of globs, a `tests` list of only `!` exclusions, the removed top-level `limits` key, an agents entry that is a bare `StepExecutor` (the message names `{ executor }`) or a model instance (the message names `{ model }`), an unknown agent key (the message names the nearest one), the removed agent keys `timeout` and `maxTurns` (the message names `judgmentTimeout` and `maxModelCalls`), `system` or `tools` beside `executor`, a `tools` entry `defineTool` did not build or that takes a built-in tool name (a grammar verb, `complete_step`, `locate`, `start_recording`, `stop_recording`, or `report_finding`), `createToolLoopExecutor({ maxTurns })`, `reuseExisting` on an app whose `app.url` asks for a free port, or a port-0 `app.url` with no `app.command` to serve it | configuration | 2 |
|
|
88
89
|
| `CONFIG_NOT_DETERMINISTIC` | A worker re-resolved the config module and got a different digest than the runner | configuration | 2 |
|
|
89
90
|
| `APP_URL_REQUIRED` | A navigation or relative URL on a target that declares no `app.url`, or an `app.command` with no `app.url` or `app.readyUrl` to poll | configuration | 2 |
|
|
90
91
|
| `INVALID_APP_URL` | Base URL with userinfo, a query, a fragment, a non-HTTP scheme, plain HTTP on a non-loopback host, port 0 on a host other than a literal loopback address (`127.0.0.1`, `[::1]`), a loopback address the run could not bind a free port on, or unparseable | configuration | 2 |
|
|
@@ -104,24 +105,26 @@ These appear in terminal output and in the report as plain string codes.
|
|
|
104
105
|
| `NO_LAST_RUN` | `--last-failed` found no `report.json` where the run writes it, or a file there that is not a `report-1` document. Run once without the flag first. | configuration | 2 |
|
|
105
106
|
| `ONLY_IN_CI` | A focused test reached a CI run | configuration | 2 |
|
|
106
107
|
| `COLLECTION_ERROR` | Registration outside collection, an invalid title, a non-function body, an async `describe` body, nested serial groups, an option forbidden inside a serial group, a setup test inside `describe`, an invalid session name, a malformed test option such as `tags` that is not a list of distinct names, a test or `describe` option key the call does not take (the message names the nearest one), a test file that fails to import (in a run with no positionals, or a file the positionals selected), a session that a selected test consumes or `e2e explore --session` names but no collected setup test produces (the message lists the declared sessions), a consumed session on a target whose engine has no `state` capability to save it, or a positional path outside the project root | configuration | 2 |
|
|
107
|
-
| `POLICY_DENIED` | A denied destination (`
|
|
108
|
+
| `POLICY_DENIED` | A denied destination (a navigation scheme other than `http:` or `https:`, `about:blank` aside, a malformed URL), a forbidden fill (an undeclared secret, a non-editable sink, a mismatched purpose, a `Secret` reaching a JSON sink), or tainted pixels (`app.screenshot()` after a secret fill, `vision: 'only'` with pixels withheld, an `act` or `vision: true` step whose semantic capture timed out before the step did); see the [security model](/security). An unknown tool or an undeclared field goes back to the model as the call's result, never this code | configuration | 2 |
|
|
108
109
|
| `SESSION_MISMATCH` | A stored session does not match the current run, target, engine, platform, or app identity | configuration | 2 |
|
|
109
110
|
| `SESSION_EXPIRED` | A stored session is past its expiry at restore time (configuration, exit 2), or a setup test saved state whose expiry is not after its creation (test, exit 1) | configuration or test | 2 or 1 |
|
|
110
111
|
| `SESSION_INVALID` | A stored session failed authentication. Tampered or corrupt. | configuration | 2 |
|
|
111
112
|
| `SESSION_UNAVAILABLE` | A test requires a session no selected setup test produced | configuration | 2 |
|
|
112
113
|
| `SESSION_CONTRACT` | A setup test saved a session twice, saved an undeclared name, or finished without saving a declared one | test | 1 |
|
|
113
|
-
| `TEST_TIMEOUT` | The attempt exceeded the resolved test `timeout`. The body is not stopped: the harness operation it is in is cancelled and its fixtures die with the attempt, so it cannot reach the next test | test | 1 |
|
|
114
|
-
| `STEP_NOT_AWAITED` | The test body settled while a step it started was still running: a step call without `await`. The step is failed at the line of the call and cancelled; the attempt fails with this code when the body returned, or keeps the body's own error and records this one under `secondaryErrors` when the body threw | test | 1 |
|
|
114
|
+
| `TEST_TIMEOUT` | The attempt exceeded the resolved test `timeout`. The body is not stopped: the harness operation it is in is cancelled and its fixtures die with the attempt, so it cannot reach the next test. A test that blocks the event loop gets its worker killed once `cleanupTimeout` (at least 5 s) passes after the timeout; see [Hooks](/reference/test#hooks) | test | 1 |
|
|
115
|
+
| `STEP_NOT_AWAITED` | The test body settled while a step it started was still running: a step call without `await`. The step is failed at the line of the call and cancelled; the attempt fails with this code when the body returned, or keeps the body's own error and records this one under `secondaryErrors` when the body threw. An `expect.poll` matcher left running when the body, an `afterEach` hook, or a fixture teardown returned fails that phase the same way at the line of the call, and is cancelled; one left by a `beforeAll` or `afterAll` hook fails the hook with `HOOK_FAILED` | test | 1 |
|
|
115
116
|
| `UNHANDLED_REJECTION` | A promise rejected with no test running. An unhandled rejection is charged to the test running when it surfaces, since Node cannot say which test created the promise; between tests it is this run error, naming the last test that finished in the worker as the likely source. Before any test has finished in the worker it ends the worker instead: the rejection is the run error and the test on its way fails with `WORKER_CRASH`. The body that left it keeps running like one past `TEST_TIMEOUT` | test | 1 |
|
|
116
117
|
| `HOOK_FAILED` | `beforeAll` or `afterAll` threw. Skips the suite's tests. | test | 1 |
|
|
117
118
|
| `LAUNCH_TIMEOUT` | Engine init or attempt start exceeded `config.launchTimeout` | infrastructure | 3 |
|
|
118
119
|
| `CLEANUP_TIMEOUT` | Engine attempt end exceeded `config.cleanupTimeout`, or a route or dialog handler the engine ran for the test was still running when the attempt settled | infrastructure | 3 |
|
|
119
|
-
| `TRACE_WITHHELD` | A trace from
|
|
120
|
+
| `TRACE_WITHHELD` | A trace from a session that knows a secret value could not be rewritten, so it was deleted, or the engine returned an archive path outside the attempt's artifact directory, so nothing was registered. Recorded in the attempt's `secondaryErrors`; never fails the attempt. | infrastructure | 3 |
|
|
120
121
|
| `BROWSER_INSTALL_FAILED` | Browser provisioning failed | infrastructure | 3 |
|
|
121
122
|
| `WORKER_INIT_FAILED` | A worker process never became ready | infrastructure | 3 |
|
|
122
|
-
| `WORKER_EXIT` | A worker exited unexpectedly while running a file: the test in flight fails with `WORKER_CRASH` and the file's other tests are skipped. An unhandled rejection is charged to the test running when it surfaces, or recorded as `UNHANDLED_REJECTION` between tests; before any test has finished there it ends the worker, and the rejection itself is the run error, not this one | infrastructure | 3 |
|
|
123
|
+
| `WORKER_EXIT` | A worker exited unexpectedly while running a file: the test in flight fails with `WORKER_CRASH` and the file's other tests are skipped. Attempts that finished before the exit stay in the report, a retry's earlier attempts and a serial group's finished members included, and the crash is recorded as the attempt after them, or on the last attempt when it came after it (an `afterAll`). An unhandled rejection is charged to the test running when it surfaces, or recorded as `UNHANDLED_REJECTION` between tests; before any test has finished there it ends the worker, and the rejection itself is the run error, not this one | infrastructure | 3 |
|
|
123
124
|
| `WORKER_CRASH` | A worker exited during a specific test | infrastructure | 3 |
|
|
124
125
|
| `WORKER_ENTRY_MISSING` | The worker entry module could not be resolved. Packaging fault. | infrastructure | 3 |
|
|
126
|
+
| `NODE_UNSUPPORTED` | The runtime cannot run e2e's TypeScript loader: a Node.js outside `^22.22.3 \|\| >=24.8.0`, Bun (no `module.registerHooks`), or Deno. The CLI checks this before it starts and exits 2 with the same sentence; the loader checks it again where it registers its hooks, for a host that runs e2e's runner without the CLI | infrastructure | 3 |
|
|
127
|
+
| `TYPESCRIPT_COMPILER_UNAVAILABLE` | oxc, which compiles e2e's TypeScript, could not be loaded; the message gives the cause. Most often its native bindings for this platform (`@oxc-transform/binding-<platform>-<arch>`, `@oxc-parser/binding-<platform>-<arch>`) are not installed. npm leaves it out with `--omit=optional`, or when `package-lock.json` was written on another platform (npm/cli#4828): install again without `--omit=optional`, or delete `package-lock.json` and `node_modules` and run `npm install` | infrastructure | 3 |
|
|
125
128
|
| `ENGINE_FAILURE` | Any engine fault with no more specific mapping | infrastructure | 3 |
|
|
126
129
|
| `WORKER_PROTOCOL` | A worker reported a unit other than the one it was running, or started an attempt while another was still running | infrastructure | 3 |
|
|
127
130
|
| `SESSION_KEY_INVALID` | A worker received a session key that is not 32 bytes. Packaging or IPC fault. | infrastructure | 3 |
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: expect
|
|
3
3
|
description: "Locator assertions, fixture assertions, and plain value matchers."
|
|
4
|
+
icon: "circle-check"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
Use `expect(locator)` to wait for an app condition and `expect(value)` to
|
|
@@ -52,14 +53,17 @@ before passing its result to `expect`.
|
|
|
52
53
|
| Default timeout | `config.assertionTimeout`, 5000 ms |
|
|
53
54
|
| Override | `{ timeout }` in milliseconds, on every locator and `browser` matcher |
|
|
54
55
|
| Unknown option | `INVALID_ARGUMENT` before the first read, from JavaScript too |
|
|
55
|
-
| Negation grace window | A negated matcher passes after 1000 ms of continuous truth, or
|
|
56
|
+
| Negation grace window | A negated matcher passes after 1000 ms of continuous truth, or at the deadline when `{ timeout }` is shorter than that and the negation held for the whole budget. The window counts from the moment the first read that saw the negation started, so a slow read counts toward it |
|
|
56
57
|
| Failure code | `ASSERTION_FAILED`, exit code 1 |
|
|
57
58
|
|
|
58
59
|
Locator strictness: except for `toHaveCount` and the list form of
|
|
59
60
|
`toHaveText` and `toContainText`, a positive matcher requires exactly one
|
|
60
61
|
match; zero matches keep polling until the timeout, and multiple matches fail
|
|
61
|
-
immediately with `LOCATOR_AMBIGUOUS`. `toBeHidden`,
|
|
62
|
-
|
|
62
|
+
immediately with `LOCATOR_AMBIGUOUS`. `toBeHidden`, `toBeVisible({ visible:
|
|
63
|
+
false })`, `toBeAttached({ attached: false })`, negated visibility, and
|
|
64
|
+
negated `toBeAttached` accept zero matches; for them, and for
|
|
65
|
+
`waitFor({ state: 'detached' })` and `'hidden'`, a locator under a frame that
|
|
66
|
+
is not in the document is zero matches rather than a wait for the frame.
|
|
63
67
|
|
|
64
68
|
Text comparison is normalized: leading and trailing whitespace is trimmed and
|
|
65
69
|
each run of whitespace collapses to one space. `toHaveText` and
|
|
@@ -201,8 +205,8 @@ Every matcher call is a recorded `expect.<matcher>` assertion step.
|
|
|
201
205
|
export interface BrowserExpectation {
|
|
202
206
|
/** Inverts the matcher. */
|
|
203
207
|
readonly not: BrowserExpectation;
|
|
204
|
-
/** Waits for the current URL to match. */
|
|
205
|
-
toHaveURL(expected: string | RegExp, options?: { timeout?: number }): Promise<void>;
|
|
208
|
+
/** Waits for the current URL to match; `ignoreCase` compares a string case-insensitively and adds the `i` flag to a RegExp, `false` removes it. */
|
|
209
|
+
toHaveURL(expected: string | RegExp, options?: { ignoreCase?: boolean; timeout?: number }): Promise<void>;
|
|
206
210
|
/** Waits for the current title to match. */
|
|
207
211
|
toHaveTitle(expected: TextMatch, options?: { timeout?: number }): Promise<void>;
|
|
208
212
|
/** Waits for the target's `class` attribute to match: a string is the whole normalized class list, a RegExp is tested against it. */
|
|
@@ -212,9 +216,10 @@ export interface BrowserExpectation {
|
|
|
212
216
|
|
|
213
217
|
A relative `toHaveURL` string resolves against the base URL and matches
|
|
214
218
|
exactly after normalization; a `RegExp` tests the full URL. Anything else,
|
|
215
|
-
such as a predicate function, is `INVALID_ARGUMENT`.
|
|
216
|
-
`
|
|
217
|
-
`
|
|
219
|
+
such as a predicate function, is `INVALID_ARGUMENT`. `toHaveURL` also takes
|
|
220
|
+
Playwright's `ignoreCase`, with the locator matchers' semantics. Otherwise each
|
|
221
|
+
`browser` matcher takes `timeout` only; any other key, such as `ignoreCase` on
|
|
222
|
+
`toHaveTitle`, which Playwright does not take either, is `INVALID_ARGUMENT`.
|
|
218
223
|
|
|
219
224
|
`toHaveClass` polls for the node too: a target that matches nothing yet is one
|
|
220
225
|
more poll, not a failure, until the assertion timeout runs out. An ambiguous
|
|
@@ -326,8 +331,11 @@ side is refused.
|
|
|
326
331
|
`toHaveProperty` walks a dotted path or an array of keys. Use the array form
|
|
327
332
|
when a key contains a dot, or to index an array by number
|
|
328
333
|
(`['lines', 0, 'sku']`). Inherited properties count; a property holding
|
|
329
|
-
`undefined` exists.
|
|
330
|
-
|
|
334
|
+
`undefined` exists. A string, number, or boolean on the path is read through
|
|
335
|
+
its wrapper object, as in Jest, so `toHaveProperty('label.length', 3)` and
|
|
336
|
+
`toHaveProperty(['label', 0], 'a')` reach into a string; `null` and
|
|
337
|
+
`undefined` have no properties. With a second argument the value at the path
|
|
338
|
+
must equal it by `toEqual` rules.
|
|
331
339
|
|
|
332
340
|
### Matching a schema
|
|
333
341
|
|
|
@@ -363,7 +371,7 @@ The unit-runner matchers (`toThrow`, `toBeInstanceOf`, `toBeNaN`,
|
|
|
363
371
|
`toStrictEqual`, `resolves`, `rejects`, `expect.extend`) are not part of
|
|
364
372
|
`expect`. An end-to-end test checks what the app shows and what its API
|
|
365
373
|
returns; the value matchers cover that, see
|
|
366
|
-
[
|
|
374
|
+
[API](/api-testing). Code-level
|
|
367
375
|
checks belong in the unit runner.
|
|
368
376
|
|
|
369
377
|
Pass a second argument to describe the check in its failure message:
|
|
@@ -455,11 +463,15 @@ ASSERTION_FAILED: 2 soft assertions failed
|
|
|
455
463
|
A soft locator or `browser` matcher is still a recorded `expect.<matcher>` step,
|
|
456
464
|
marked failed, so the report shows where in the flow it landed. A soft value
|
|
457
465
|
matcher is not a step, the same as a hard one; its failure appears in the
|
|
458
|
-
|
|
466
|
+
attempt's error records.
|
|
459
467
|
|
|
460
468
|
When the body throws on its own, or times out, that error stays the
|
|
461
469
|
attempt's verdict and the soft failures are recorded beside it under
|
|
462
|
-
`secondaryErrors`. A body that skips itself with `test.skip(...)`
|
|
470
|
+
`secondaryErrors`. A body that skips itself with `test.skip(...)` keeps them
|
|
471
|
+
there too. The CLI shows `Skipped After Failure`; set
|
|
472
|
+
[`failOnSkippedFailure: true`](/reference/config) to make that run fail with
|
|
473
|
+
exit code 1. Without that option, the test stays skipped and does not affect
|
|
474
|
+
the exit code.
|
|
463
475
|
|
|
464
476
|
A soft `toMatchSchema` returns the schema's output when the value matches
|
|
465
477
|
and `undefined` after a kept failure, and its type says so.
|
|
@@ -544,3 +556,14 @@ await expect.poll(() => readStatus(), { message: 'the batch never finished' }).n
|
|
|
544
556
|
|
|
545
557
|
`expect.poll` is not recorded as a report step; a timeout surfaces only as
|
|
546
558
|
the test's failure.
|
|
559
|
+
|
|
560
|
+
Await every poll. A poll belongs to the phase that started it: the test body
|
|
561
|
+
(with its fixtures and `beforeEach` hooks), one `afterEach` hook or fixture
|
|
562
|
+
teardown, or one `beforeAll` or `afterAll` hook. When that phase returns while
|
|
563
|
+
the poll still runs, the poll is cancelled and the phase fails with
|
|
564
|
+
`STEP_NOT_AWAITED` at the line of the call, whether or not the poll would
|
|
565
|
+
have passed. It never fails a later test: a poll started by work its phase
|
|
566
|
+
left running, such as a body past its timeout, is cancelled before its first
|
|
567
|
+
read. A body that threw keeps its own
|
|
568
|
+
error and records this one under `secondaryErrors`, and a suite hook reports
|
|
569
|
+
it as `HOOK_FAILED`.
|
package/docs/reference/mcp.mdx
CHANGED
|
@@ -1,19 +1,20 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: MCP server (e2e mcp)
|
|
3
3
|
description: Let a coding agent inspect and interact with your app over MCP.
|
|
4
|
+
icon: "plug"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
<CodeGroup>
|
|
7
8
|
```bash npm
|
|
8
|
-
npx e2e mcp [--config <path>] [--target <name>] [--
|
|
9
|
+
npx e2e mcp [--config <path>] [--target <name>] [--headed] [--max-sessions <n>]
|
|
9
10
|
```
|
|
10
11
|
|
|
11
12
|
```bash pnpm
|
|
12
|
-
pnpm exec e2e mcp [--config <path>] [--target <name>] [--
|
|
13
|
+
pnpm exec e2e mcp [--config <path>] [--target <name>] [--headed] [--max-sessions <n>]
|
|
13
14
|
```
|
|
14
15
|
|
|
15
16
|
```bash bun
|
|
16
|
-
bunx e2e mcp [--config <path>] [--target <name>] [--
|
|
17
|
+
bunx e2e mcp [--config <path>] [--target <name>] [--headed] [--max-sessions <n>]
|
|
17
18
|
```
|
|
18
19
|
</CodeGroup>
|
|
19
20
|
|
|
@@ -64,21 +65,32 @@ Start the server in the project directory. Config discovery works like
|
|
|
64
65
|
| --- | --- |
|
|
65
66
|
| `--config <path>` | The config every session loads unless `open_session` names one. Configs are reloaded on every `open_session`, with every file they import by path. |
|
|
66
67
|
| `--target <name>` | The target every session opens on. Otherwise `open_session` names one, or the only target is used; several targets and no name is `TARGET_REQUIRED`. |
|
|
67
|
-
| `--
|
|
68
|
+
| `--headed` | Show the browser or simulator, when the engine supports it, in every session whose `open_session` does not set `headed`. Sessions are headless by default, as in `e2e run`. |
|
|
68
69
|
| `--max-sessions <n>` | How many sessions may be open at once, 1 through 16; default 4. Each session is its own browser or device. |
|
|
69
70
|
|
|
70
71
|
Operator output goes to stderr, prefixed `e2e mcp:`. Stdout carries only the
|
|
71
|
-
protocol
|
|
72
|
+
protocol. What your code writes through `console`, `process.stdout`, or
|
|
73
|
+
`process.stderr` (the config's top-level code, project tools) goes to stderr
|
|
74
|
+
with secret values redacted. Output printed while a config loads appears once
|
|
75
|
+
it has loaded; if the load fails, that output is withheld, since the config's
|
|
76
|
+
secrets are unknown. An error nobody caught is logged redacted and shuts the
|
|
77
|
+
server down. A child process that inherits the server's stdio writes past
|
|
78
|
+
this.
|
|
72
79
|
|
|
73
80
|
## Tools
|
|
74
81
|
|
|
75
82
|
| Tool | Arguments | Result |
|
|
76
83
|
| --- | --- | --- |
|
|
77
|
-
| `open_session` | `target` (optional), `config` (optional path) | Loads the config, starts the target's app command, boots the engine, starts one attempt, opens the app URL when the engine can navigate. Returns the session id, the catalog, and the first observation. |
|
|
84
|
+
| `open_session` | `target` (optional), `config` (optional path), `headed` (optional boolean) | Loads the config, starts the target's app command, boots the engine, starts one attempt, opens the app URL when the engine can navigate. Returns the session id, the catalog, and the first observation. `headed: true` shows this session's browser or simulator, `false` hides it; without it the session follows `--headed`. |
|
|
78
85
|
| `tools` | `tool` (optional), `session` (required while several are open) | The catalog, one line per tool. With `tool`, its full description and argument schema. |
|
|
79
86
|
| `call` | `tool`, `args` (optional), `session` (required while several are open) | Runs one catalog tool. Arguments are validated against the tool's closed schema first: a wrong type or an argument the tool does not declare is `INVALID_ARGUMENT`, and nothing runs. An action that fails is an error result that still carries the screen it re-observed, and leads with the tool, its target, and the code: `navigate file:///etc/passwd failed: POLICY_DENIED: forbidden URL scheme: file:`. |
|
|
80
87
|
| `close_session` | `session` (required while several are open) | Saves a recording still running, ends the attempt, disposes the engine, stops the app processes the session started once no other session uses them. |
|
|
81
88
|
|
|
89
|
+
Project tools honor their AI SDK `toModelOutput` renderer, including when
|
|
90
|
+
`execute` returns a string. Rendered `error-text` and `error-json` outputs
|
|
91
|
+
are MCP error results (`isError: true`) so the coding agent can distinguish
|
|
92
|
+
a failed tool call from a successful one.
|
|
93
|
+
|
|
82
94
|
## Several sessions
|
|
83
95
|
|
|
84
96
|
Several sessions can be open at once, up to `--max-sessions`, each on its
|
|
@@ -183,8 +195,9 @@ Sessions enforce what tests enforce. Node ids are valid only against the
|
|
|
183
195
|
newest observation. A secret fills only through `type_secret`; once one has
|
|
184
196
|
been filled, `screenshot` and the point tools (`tap_at`, `hover_at`,
|
|
185
197
|
`type_at`, `press_at`, `select_at`) stay in the catalog but answer
|
|
186
|
-
`PIXEL_TAINTED` for the rest of the session. Every text a
|
|
187
|
-
an action's result, or
|
|
198
|
+
`PIXEL_TAINTED` for the rest of the session. Every text a tool returns, a node `locate` found,
|
|
199
|
+
an action's result, an `open_session` or `close_session` failure, or a cleanup line, has the
|
|
200
|
+
config's secret values redacted once the config has loaded, as do the server's own log lines. Nothing a
|
|
188
201
|
session does is recorded as a test result or into the replay cache; its
|
|
189
202
|
artifacts land under `<output>/artifacts/<target>/sessions/`, its videos under
|
|
190
203
|
`<output>/videos/`. A later `e2e run` clears `<output>/artifacts`; the
|
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Mobile
|
|
3
3
|
description: Mobile engine options and the device fixture for iOS and Android.
|
|
4
|
+
icon: "mobile-screen"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
|
-
`@e2e-dev/mobile` drives iOS simulators
|
|
7
|
+
`@e2e-dev/mobile` drives iOS simulators, Android emulators, and connected
|
|
8
|
+
phones through
|
|
7
9
|
[agent-device](https://github.com/callstack/agent-device). `agent`, `app`,
|
|
8
10
|
`screen`, and `expect` work as they do everywhere; this page covers what is
|
|
9
11
|
specific to a device target. Import `test` from the package to have `device`
|
|
@@ -79,8 +81,9 @@ export interface MobileOptions {
|
|
|
79
81
|
/** Platform the target's device runs. */
|
|
80
82
|
readonly platform: MobilePlatform;
|
|
81
83
|
/**
|
|
82
|
-
*
|
|
83
|
-
* serial (`emulator-5554`)
|
|
84
|
+
* Device to use: a simulator or emulator by name, simulator UDID, or
|
|
85
|
+
* emulator serial (`emulator-5554`), or a connected phone by name. A list
|
|
86
|
+
* is a pool: the
|
|
84
87
|
* engine declares one worker per entry and worker slot `n` drives the
|
|
85
88
|
* `n`th, so `workers` at or above the pool size runs the target's files
|
|
86
89
|
* across every device at once. Omitted, the pool is every booted device of
|
|
@@ -278,8 +281,8 @@ export interface Device {
|
|
|
278
281
|
* `relaunch` restarts it fresh. `launchArguments` and `permissions` apply
|
|
279
282
|
* to this launch alone; the target's configured ones apply to a `relaunch`
|
|
280
283
|
* of the pinned app, and to no foreground-only open. A link is
|
|
281
|
-
* `INVALID_ARGUMENT` (`file:`, `data:`,
|
|
282
|
-
* `POLICY_DENIED`): `openLink` opens one.
|
|
284
|
+
* `INVALID_ARGUMENT` (`file:`, `data:`, `javascript:`, `view-source:`,
|
|
285
|
+
* `blob:`, and `filesystem:` are `POLICY_DENIED`): `openLink` opens one.
|
|
283
286
|
*/
|
|
284
287
|
openApp(app: string, options?: OpenAppOptions): Promise<void>;
|
|
285
288
|
/**
|
|
@@ -289,8 +292,8 @@ export interface Device {
|
|
|
289
292
|
* a web link the app does not claim goes to Safari, and the next
|
|
290
293
|
* observation brings the app back. Android starts the link on that
|
|
291
294
|
* package. Without an app, Android lets the OS route the link; iOS needs
|
|
292
|
-
* one (`INVALID_ARGUMENT` without). `file:`, `data:`,
|
|
293
|
-
* links are `POLICY_DENIED`.
|
|
295
|
+
* one (`INVALID_ARGUMENT` without). `file:`, `data:`, `javascript:`,
|
|
296
|
+
* `view-source:`, `blob:`, and `filesystem:` links are `POLICY_DENIED`.
|
|
294
297
|
*/
|
|
295
298
|
openLink(url: string, options?: { app?: string }): Promise<void>;
|
|
296
299
|
/**
|
|
@@ -426,7 +429,10 @@ session opened. Android reads the device's foreground activity.
|
|
|
426
429
|
`installApp` installs a build from inside a test or a fixture: the run's build once per device (no path installs `app.appPath`), or another build for an upgrade path. A plain
|
|
427
430
|
install keeps the app's data; `reinstall: true` removes the app named by
|
|
428
431
|
`app` (default: the pinned app) first. It resolves to the id to `openApp`
|
|
429
|
-
the build by
|
|
432
|
+
the build by: the bundle id or package agent-device read from the build, else
|
|
433
|
+
the `app` passed in. With no reported id and no `app`, the install is
|
|
434
|
+
`ENGINE_FAILURE` instead of a file path posing as an id; see
|
|
435
|
+
[Troubleshooting](/mobile#troubleshooting).
|
|
430
436
|
|
|
431
437
|
```ts
|
|
432
438
|
const { app } = await device.installApp('./build/MyApp.app', { reinstall: true });
|
|
@@ -444,9 +450,9 @@ name (`Settings`); `relaunch: true` restarts it fresh, and `launchArguments`
|
|
|
444
450
|
and `permissions` apply to that launch alone (the target's own apply to a
|
|
445
451
|
`relaunch` of the pinned app). It takes no link:
|
|
446
452
|
agent-device opens any `scheme:rest` string as a URL, so a link is refused
|
|
447
|
-
before the device sees it. A `file:`, `data:`,
|
|
448
|
-
`
|
|
449
|
-
`INVALID_ARGUMENT` pointing at `openLink`.
|
|
453
|
+
before the device sees it. A `file:`, `data:`, `javascript:`,
|
|
454
|
+
`view-source:`, `blob:`, or `filesystem:` link is `POLICY_DENIED`; any other
|
|
455
|
+
link is `INVALID_ARGUMENT` pointing at `openLink`.
|
|
450
456
|
|
|
451
457
|
`openLink` opens a deep link or a web link into `app` (default: the pinned
|
|
452
458
|
app), for magic-link sign-in and deep-link routes, and the session observes
|
|
@@ -456,8 +462,13 @@ observation brings the app back. Android starts the link on that package.
|
|
|
456
462
|
Without an app, Android lets the OS route the link, to a verified app link
|
|
457
463
|
owner or the browser, and the session follows the package that took it. iOS
|
|
458
464
|
needs an app: an open bound to none leaves no app session to observe, so it
|
|
459
|
-
is `INVALID_ARGUMENT`. `file:`, `data:`,
|
|
460
|
-
`
|
|
465
|
+
is `INVALID_ARGUMENT`. `file:`, `data:`, `javascript:`, `view-source:`,
|
|
466
|
+
`blob:`, and `filesystem:` links are `POLICY_DENIED`. A device link may use
|
|
467
|
+
any app's custom scheme, so the rule refuses these schemes rather than
|
|
468
|
+
admitting http(s) alone as the web does. A string that does not parse as an
|
|
469
|
+
absolute URL is `INVALID_ARGUMENT` too, and its message names no part of the
|
|
470
|
+
input, which may carry a magic-link token in its query, path, or userinfo.
|
|
471
|
+
The report shows the link cut before its
|
|
461
472
|
query and fragment, where a magic link carries its token. The replay cache
|
|
462
473
|
anchors on the location the link lands on: a step recorded after `openLink`
|
|
463
474
|
replays when the link lands on the same screen again.
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Reporters
|
|
3
3
|
description: Receive run events and publish results with a custom reporter.
|
|
4
|
+
icon: "file-lines"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
Add reporter objects to `reporters` in `e2e.config.ts`, alongside the
|
|
@@ -171,7 +172,7 @@ prints. All reporters run concurrently, each with a one-minute deadline.
|
|
|
171
172
|
A timeout, exception, or invalid return value is reported on stderr without
|
|
172
173
|
changing the run's status, exit code, or report.
|
|
173
174
|
|
|
174
|
-
One Ctrl-C still lets reporters finish. A second Ctrl-C stops waiting for them.
|
|
175
|
+
One Ctrl-C still lets reporters finish. A second Ctrl-C stops waiting for them, except the built-in `junit` and `markdown`, which still write their files so none of the previous run's is left beside the new report.
|
|
175
176
|
|
|
176
177
|
The second argument is an `AbortSignal` that aborts when the budget runs out
|
|
177
178
|
or the run is forced to stop. Hand it to every request and timer, or an
|
|
@@ -180,7 +181,13 @@ abandoned reporter keeps the CLI alive after the run has ended.
|
|
|
180
181
|
With `--last-failed`, `lastRun` is the report the rerun selected from. The
|
|
181
182
|
rerun's own report lists the tests it left out as unselected, so a reporter
|
|
182
183
|
that keeps one place current, the way `@e2e-dev/github` keeps a pull request
|
|
183
|
-
comment, folds the rerun into `lastRun` to show both runs as one.
|
|
184
|
+
comment, folds the rerun into `lastRun` to show both runs as one. A test the
|
|
185
|
+
rerun owed but another filter left out is in the rerun's `run.carried`: its
|
|
186
|
+
last result, the serial group its attempts live on, and the `beforeAll` or
|
|
187
|
+
`afterAll` failure whose scope holds it, from whichever earlier run reported
|
|
188
|
+
them. A fold shows the carried result in place of the unselected one and
|
|
189
|
+
counts a carried failure or hook error against the outcome. The artifacts
|
|
190
|
+
both reports name are on disk under `artifactsRoot`.
|
|
184
191
|
|
|
185
192
|
The rows a reporter resolves with print under the `list` summary in the same
|
|
186
193
|
layout as `Report`; the `json` reporter does not carry them:
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: screen and Locator
|
|
3
3
|
description: Queries, refinement, actions, and reads.
|
|
4
|
+
icon: "display"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
A `Locator` stores a query and resolves it when an action, read, or assertion
|
|
@@ -107,7 +108,14 @@ await screen.getByTestId('memory-list', { visible: true }).getByRole('listitem')
|
|
|
107
108
|
```
|
|
108
109
|
|
|
109
110
|
`visible` uses the same visibility check as `toBeVisible()`, `isVisible()`,
|
|
110
|
-
and `waitFor()
|
|
111
|
+
and `waitFor()`: what renders, as Playwright reads it. `display: none`,
|
|
112
|
+
`visibility: hidden`, a box with no size, a closed `<details>` body, and
|
|
113
|
+
content under `content-visibility: hidden` are hidden; `aria-hidden` is not,
|
|
114
|
+
since a decorative spinner marked with it still paints, so
|
|
115
|
+
`toBeHidden()` on one waits until it leaves. A child that sets
|
|
116
|
+
`visibility: visible` under a hidden parent is visible. On the web, text
|
|
117
|
+
directly under a `display: contents` element with `visibility: hidden` is
|
|
118
|
+
hidden, where Playwright calls it visible. It works with scopes, `filter`, `first`, `last`, and `nth`:
|
|
111
119
|
`getByText('Save', { visible: true }).first()` is the first visible match.
|
|
112
120
|
Prefer it over `browser.locator('p:visible')`, which ties the test to CSS.
|
|
113
121
|
|
|
@@ -171,14 +179,15 @@ not the finger: `'down'` reveals what is below. `momentum` is its fling
|
|
|
171
179
|
strength, `'none'` by default: `'none'` a plain drag of half the surface,
|
|
172
180
|
`'slow'` a short fling, `'fast'` a long one. How far each carries is the
|
|
173
181
|
engine's.
|
|
174
|
-
`{ from, to }` is a swipe along a path between two viewport points:
|
|
175
|
-
swipe on a device, a pointer drag on a document platform, which is
|
|
176
|
-
reveals a row action, dismisses a sheet, or drags a card between
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
`
|
|
182
|
+
`{ from, to, duration? }` is a swipe along a path between two viewport points:
|
|
183
|
+
a touch swipe on a device, a pointer drag on a document platform, which is
|
|
184
|
+
what reveals a row action, dismisses a sheet, or drags a card between
|
|
185
|
+
columns. `duration` sets how long the gesture takes in milliseconds; omitted
|
|
186
|
+
leaves the duration to the engine's default. Uses `config.actionTimeout`.
|
|
187
|
+
Throws `ACTION_FAILED` when the gesture is rejected, `INVALID_ARGUMENT` for
|
|
188
|
+
a mix of the two shapes, an out-of-range duration, or a coordinate off the
|
|
189
|
+
plane, and `UNSUPPORTED_CAPABILITY` when the engine does not list `swipe` in
|
|
190
|
+
its actions (a direction) or `swipeTo` in its pointer actions (a path).
|
|
182
191
|
|
|
183
192
|
```ts
|
|
184
193
|
await screen.swipe({ direction: 'up', momentum: 'fast' });
|
|
@@ -187,6 +196,7 @@ const row = await screen.getByText('Design review').boundingBox();
|
|
|
187
196
|
await screen.swipe({
|
|
188
197
|
from: { x: row.x + row.width - 10, y: row.y + row.height / 2 },
|
|
189
198
|
to: { x: row.x + 10, y: row.y + row.height / 2 },
|
|
199
|
+
duration: 300,
|
|
190
200
|
});
|
|
191
201
|
```
|
|
192
202
|
|
|
@@ -292,8 +302,12 @@ export interface Locator extends Screen {
|
|
|
292
302
|
all(): Promise<Locator[]>;
|
|
293
303
|
/** Reads the normalized text of every current match, without auto-waiting; empty when nothing matches. */
|
|
294
304
|
allTextContents(): Promise<string[]>;
|
|
295
|
-
/**
|
|
296
|
-
|
|
305
|
+
/**
|
|
306
|
+
* Waits for the requested locator state, `visible` by default: `attached`
|
|
307
|
+
* for one match, visible or not; `detached` for none; `hidden` for none or
|
|
308
|
+
* a hidden one.
|
|
309
|
+
*/
|
|
310
|
+
waitFor(options?: { state?: 'attached' | 'detached' | 'visible' | 'hidden'; timeout?: number }): Promise<void>;
|
|
297
311
|
/** Adds deterministic locator filters. */
|
|
298
312
|
filter(options: { hasText?: TextMatch; has?: Locator }): Locator;
|
|
299
313
|
/** Selects the first current match. */
|
|
@@ -382,7 +396,11 @@ await screen.getByLabel('City').pressSequentially('War', { delay: 50 });
|
|
|
382
396
|
## Locator reads
|
|
383
397
|
|
|
384
398
|
`waitFor` defaults `state` to `'visible'` and `timeout` to
|
|
385
|
-
`config.actionTimeout`, and throws `LOCATOR_NOT_FOUND` on timeout.
|
|
399
|
+
`config.actionTimeout`, and throws `LOCATOR_NOT_FOUND` on timeout. `'attached'`
|
|
400
|
+
waits for one match, hidden or shown, `'detached'` for none, and `'hidden'` for
|
|
401
|
+
a hidden match or none. Several matches are `LOCATOR_AMBIGUOUS` in every
|
|
402
|
+
state. Any other state, or an option key besides `state` and `timeout`, is
|
|
403
|
+
`INVALID_ARGUMENT` before anything is resolved.
|
|
386
404
|
|
|
387
405
|
`textContent` and `allTextContents` return the node's rendered text with
|
|
388
406
|
whitespace collapsed to single spaces. On the web that is what `innerText`
|
|
@@ -399,6 +417,12 @@ a heading; it does not throw.
|
|
|
399
417
|
`toHaveText`, `toContainText`, and `toHaveAttribute`.
|
|
400
418
|
`isEnabled`, `isDisabled`, `isChecked`, and `boundingBox` are allowed there.
|
|
401
419
|
|
|
420
|
+
`boundingBox()` is `null` for a node that lays out no box (`display: none`
|
|
421
|
+
on it or an ancestor), and a box for one `visibility: hidden` keeps.
|
|
422
|
+
`inputValue()` and `toHaveValue` on a checkbox or radio read its value
|
|
423
|
+
attribute, `on` when it has none, whatever its checked state; `isChecked()`
|
|
424
|
+
reads the state.
|
|
425
|
+
|
|
402
426
|
`isHidden` is `!isVisible()` and `isDisabled` is `!isEnabled()`, with the
|
|
403
427
|
same strictness: `isHidden` is `true` for zero matches, `isDisabled` is
|
|
404
428
|
`LOCATOR_NOT_FOUND` there, and both are `LOCATOR_AMBIGUOUS` on several.
|
|
@@ -521,7 +545,7 @@ export interface TextMatchOptions {
|
|
|
521
545
|
/**
|
|
522
546
|
* Role query options. A role query never matches a node hidden from the
|
|
523
547
|
* accessibility tree, on every engine; `visible` (inherited) is the one knob
|
|
524
|
-
* that narrows the other query kinds the
|
|
548
|
+
* that narrows the other query kinds, to the nodes `toBeVisible()` accepts.
|
|
525
549
|
*/
|
|
526
550
|
export interface RoleOptions extends TextMatchOptions {
|
|
527
551
|
/** Accessible name filter. */
|
|
@@ -564,6 +588,7 @@ export interface SwipeOptions {
|
|
|
564
588
|
momentum?: Momentum;
|
|
565
589
|
from?: never;
|
|
566
590
|
to?: never;
|
|
591
|
+
duration?: never;
|
|
567
592
|
}
|
|
568
593
|
|
|
569
594
|
/** A swipe along a path between two viewport points: a touch swipe on a device, a pointer drag on a document platform. */
|
|
@@ -572,6 +597,8 @@ export interface SwipePathOptions {
|
|
|
572
597
|
from: Point;
|
|
573
598
|
/** Where it lifts. */
|
|
574
599
|
to: Point;
|
|
600
|
+
/** Gesture duration in milliseconds; omitted = the engine's default. */
|
|
601
|
+
duration?: number;
|
|
575
602
|
direction?: never;
|
|
576
603
|
momentum?: never;
|
|
577
604
|
}
|
|
@@ -623,8 +650,10 @@ export type ScrollDirection = 'up' | 'down' | 'left' | 'right';
|
|
|
623
650
|
export type Momentum = 'none' | 'slow' | 'fast';
|
|
624
651
|
```
|
|
625
652
|
|
|
626
|
-
A role query never matches a node hidden from the accessibility tree
|
|
627
|
-
|
|
653
|
+
A role query never matches a node hidden from the accessibility tree: one
|
|
654
|
+
that does not render, or one under `aria-hidden`. `visible` narrows the
|
|
655
|
+
other query kinds to the nodes that render, the check `toBeVisible()` makes,
|
|
656
|
+
so an `aria-hidden` node that still paints counts as visible there. See
|
|
628
657
|
[Hidden duplicates](#hidden-duplicates).
|
|
629
658
|
|
|
630
659
|
`img` is an alias, not a role: `getByRole('img', { name })` builds the same
|
|
@@ -640,7 +669,11 @@ and `<details>` are `group`, `<input type="number">` is `spinbutton`,
|
|
|
640
669
|
`<select size="3">` is a `listbox` like a `multiple` one, an `<input list>`
|
|
641
670
|
naming a `<datalist>` is a `combobox`, a `<td>` of a `role="grid"` table is a
|
|
642
671
|
`gridcell`,
|
|
643
|
-
`<thead>` and `<tbody>` are `rowgroup`, `<th scope="row">` is `rowheader
|
|
672
|
+
`<thead>` and `<tbody>` are `rowgroup`, `<th scope="row">` is `rowheader`.
|
|
673
|
+
A scope-less `<th>` beside a data cell with text or children is also a
|
|
674
|
+
`rowheader`; header-only rows keep `columnheader`. A lone scope-less header in a one-row
|
|
675
|
+
table has no header role, while `scope="col"` and `scope="colgroup"` explicitly
|
|
676
|
+
keep `columnheader`.
|
|
644
677
|
`<figure>` is `figure`, `<article>` is `article`, `<header>` and `<footer>` are
|
|
645
678
|
`banner` and `contentinfo` only outside sectioning content or an element with
|
|
646
679
|
a sectioning role (`article`, `complementary`, `main`, `navigation`, `region`),
|
package/docs/reference/test.mdx
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: test
|
|
3
3
|
description: Tests, setup tests, groups, hooks, options, and fixtures.
|
|
4
|
+
icon: "flask-vial"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
Declare tests, groups, and hooks synchronously while the test file is
|
|
@@ -97,16 +98,23 @@ group titles and then the test's own, are a `COLLECTION_ERROR`.
|
|
|
97
98
|
|
|
98
99
|
## Members
|
|
99
100
|
|
|
100
|
-
A skipped test
|
|
101
|
+
A skipped test does not affect the exit code by default. With `test.only`, other tests
|
|
101
102
|
are reported as `filtered`, and a CI run fails with `ONLY_IN_CI`.
|
|
102
103
|
|
|
103
104
|
Call `test.skip(condition, reason)` inside a test for conditions you learn at
|
|
104
105
|
runtime, such as a disabled feature flag. When the condition is true, the call
|
|
105
106
|
throws: the body stops at that line, as Playwright's does, and the test is
|
|
106
107
|
reported `skipped` with `reason`, with no retry. Nothing after the call runs.
|
|
107
|
-
Steps before it stay in the report as they ran
|
|
108
|
-
|
|
109
|
-
|
|
108
|
+
Steps before it stay in the report as they ran. Soft failures before the call
|
|
109
|
+
remain in the attempt's `secondaryErrors`, and the CLI lists the test under
|
|
110
|
+
`Skipped After Failure`. A failed attempt followed by a skip on retry appears
|
|
111
|
+
there too, with its original error. Hooks and fixtures still run their teardown.
|
|
112
|
+
When the condition is a precondition, call it first, before any step.
|
|
113
|
+
|
|
114
|
+
Set [`failOnSkippedFailure: true`](/reference/config) to fail the run with exit
|
|
115
|
+
code 1 when a runtime skip follows a failure in that test. The test remains
|
|
116
|
+
`skipped` with its reason, and the skip does not trigger another retry. Clean
|
|
117
|
+
skips and retries that pass are unaffected. The default is `false`.
|
|
110
118
|
|
|
111
119
|
A setup test cannot skip because other tests depend on its sessions; the
|
|
112
120
|
call fails with `INVALID_ARGUMENT`. Outside a test body, the conditional
|
|
@@ -185,6 +193,19 @@ for every scope whose `beforeAll` started.
|
|
|
185
193
|
its own `cleanupTimeout` budget with working fixtures, so teardown can still
|
|
186
194
|
drive the app after the body timed out.
|
|
187
195
|
|
|
196
|
+
A test that blocks its worker's event loop (`while (true) {}` in a fixture,
|
|
197
|
+
`beforeEach`, the body, or `afterEach`) stops every timer in that worker, so
|
|
198
|
+
the worker cannot time it out. A `beforeAll` or `afterAll` that blocks still
|
|
199
|
+
hangs the run. When the worker has not responded `cleanupTimeout` (at least 5 s) after
|
|
200
|
+
the test timeout, the runner kills it. That attempt is reported `timed-out`
|
|
201
|
+
with `TEST_TIMEOUT` and no artifacts, after the attempts before it, and is
|
|
202
|
+
not retried. The rest of its file runs on a new worker. In a serial group,
|
|
203
|
+
the members after it are skipped. A worker with a debugger attached when the
|
|
204
|
+
test starts (an editor's auto-attach, or `NODE_OPTIONS=--inspect=0`) is
|
|
205
|
+
never killed this way, so a breakpoint can stay paused. A programmatic run with an inline
|
|
206
|
+
config or in-memory tests runs in the runner's own process and has no worker
|
|
207
|
+
to kill: a blocked event loop hangs it.
|
|
208
|
+
|
|
188
209
|
A throw inside `beforeAll` or `afterAll` is `HOOK_FAILED` and skips the
|
|
189
210
|
suite's tests with cause `hook-failed`. Inside a serial group it skips every
|
|
190
211
|
remaining member and fails the group without a retry.
|
|
@@ -295,7 +316,13 @@ group's attempt. A retry starts fresh. Inside it, `retries`, `trace`,
|
|
|
295
316
|
`video`, `session`, `platforms`, `requires`, `skip`, `only`, and a nested
|
|
296
317
|
`serial` on a member are collection errors. Set them on the group. The group
|
|
297
318
|
records one trace and one video per group attempt, from the `trace` and
|
|
298
|
-
`video` on the serial `describe` or a group around it.
|
|
319
|
+
`video` on the serial `describe` or a group around it. A group attempt that
|
|
320
|
+
cannot start, because its file fails to load or its session fails to launch,
|
|
321
|
+
fails the first member with that error and skips the rest with cause
|
|
322
|
+
`serial-predecessor-failed`. When the run is interrupted during the launch,
|
|
323
|
+
every member skips instead. On a retry, the members keep the verdicts of
|
|
324
|
+
the attempt before it. The exit code follows the error's category, as for any
|
|
325
|
+
other test.
|
|
299
326
|
|
|
300
327
|
## Fixtures
|
|
301
328
|
|