e2e 0.17.0 → 0.18.0

package/docs/assertions.mdx

@@ -1,94 +0,0 @@
----
-title: Assertions
-description: Check the screen with agent.assert, wait for a condition, and extract data.
----
-
-An assertion checks what is on screen. The agent has three judgment calls:
-`assert` checks the screen now, `waitFor` waits until a condition holds, and
-`extract` reads structured data. When you know the exact value, use an
-`expect` [locator assertion](#locator-assertions) instead, with no model call.
-
-```ts
-await agent.assert('the dashboard shows a trial badge');
-```
-
-## `agent.assert`
-
-`assert` checks a statement against the current screen. A false statement
-fails with `ASSERTION_FAILED`. If the screen does not provide enough evidence,
-it fails with `ASSERTION_INCONCLUSIVE`.
-
-The judging model sees the statement and the current screen. It does not see
-earlier steps or the acting agent's summaries.
-
-## `agent.waitFor`
-
-`assert` checks the screen now. Use `waitFor` when the condition may become true
-later. It polls the screen and only calls the model when the observation changes.
-
-```ts
-await agent.waitFor('the export finishes and a download link appears', {
-  interval: 500,
-  timeout: 120_000,
-});
-```
-
-## `agent.extract`
-
-`extract` reads structured data from the screen. Provide a
-[Standard Schema](https://standardschema.dev) validator, such as Zod, to define
-the result.
-
-```ts
-import { expect } from 'e2e';
-import { z } from 'zod';
-
-const data = await agent.extract('every todo title and how many remain', {
-  schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }),
-});
-expect(data.titles).toContain('Buy milk');
-```
-
-<Tip>
-For model-generated descriptions, check the relevant meaning rather than the
-entire sentence. Use a [locator assertion](/locators) when the exact text matters.
-</Tip>
-
-## Judging pixels
-
-By default, judgments read a redacted text snapshot of the screen. Add
-`vision: true` for visual checks, such as charts or layout. Use
-`vision: 'only'` to judge the screenshot alone.
-
-```ts
-await agent.assert('the chart trends upward', { vision: true });
-await agent.assert('the search form is not covered by a banner', { vision: 'only' });
-```
-
-Images add input tokens. Screenshots are withheld for the rest of an attempt
-after a secret is filled. The [agent reference](/reference/agent#vision)
-lists the modes and restrictions.
-
-## Locator assertions
-
-`expect` on a locator checks an exact value and calls no model. It retries
-until the check passes or the assertion timeout runs out.
-
-```ts
-await expect(screen.getByRole('status')).toHaveText('2 remaining');
-await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible();
-```
-
-[Locators](/locators) covers queries and actions. The
-[expect reference](/reference/expect#locator-matchers) lists every matcher.
-
-## Next
-
-<CardGroup cols={2}>
-  <Card title="Locators" icon="crosshairs" href="/locators">
-    Exact actions and checks with `screen` and `expect`.
-  </Card>
-  <Card title="How agent steps work" icon="list-check" href="/agent-steps">
-    Verdicts, exit codes, and what the model sees.
-  </Card>
-</CardGroup>

package/docs/goals.mdx

@@ -1,176 +0,0 @@
----
-title: Goals
-description: Write test steps as goals with agent.act, and structure the tests around them.
----
-
-Use agent calls for goals and questions about the screen. Use
-[locators](/locators) when you know the exact action or value to check. A test
-can freely mix both.
-
-```ts title="tests/todos.e2e.ts"
-import { test, expect } from 'e2e';
-
-test('todos survive a filter round-trip', async ({ app, agent, screen }) => {
-  await app.open('/todos');
-
-  await agent.act('add todos "Buy milk" and "Walk the dog", complete the first');
-  await agent.assert('only one todo remains open');
-
-  await screen.getByRole('tab', 'Done').tap();
-  await expect(screen.getByTestId('todo')).toHaveCount(1);
-});
-```
-
-## Goals: `agent.act`
-
-Give `act` one goal. The agent chooses and performs the actions, then returns
-when the goal passes or throws when it fails. Await every agent and screen
-call, and every `expect.poll`; a test that returns while one is still running
-fails with `STEP_NOT_AWAITED`.
-
-```ts
-await agent.act('complete checkout with the test card');
-await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
-```
-
-Pass values through `params` so the agent uses your test data. For passwords,
-use a `Secret`, this prevents the model from seeing your secrets. See [Signing in](/authentication).
-
-```ts
-import { credentials } from 'e2e';
-
-const member = credentials.user('member');
-await agent.act('Sign in', {
-  params: { username: member.username, password: member.password },
-});
-```
-
-<Tip>
-**Write instructions as you would say them out loud**
-
-Name what is visible and prefer the exact wording on screen. One goal per
-call. The flow inside a goal is the model's job; the order of goals is yours.
-</Tip>
-
-For a value that changes on every run, wrap it in `unique()` so the
-[replay cache](/cache#what-must-match) can still replay the step:
-
-```ts
-import { expect, unique } from 'e2e';
-
-const email = `ada+${Date.now()}@example.test`;
-await agent.act('sign up with email {email}', { params: { email: unique(email) } });
-await expect(screen.getByText(email)).toBeVisible();
-```
-
-If a goal fails, check the report and make the instruction more specific.
-Use the labels on screen. Put vocabulary shared by several goals in the
-agent's `context`; see [Agents and personas](/agents).
-
-A step can pass, fail, or be blocked by something such as missing credentials.
-[How agent steps work](/agent-steps#how-a-step-ends) explains the verdicts and
-exit codes.
-
-`act` requests screenshots itself when it needs them, so unlike the
-[judgments](/assertions#judging-pixels) it has no `vision` option. On
-supported engines, it can tap and type at points in the image:
-
-```ts
-await agent.act('enter the code 3141 on the drawn keypad and press OK');
-```
-
-Images add input tokens. Screenshots are withheld for the rest of an attempt
-after a secret is filled. The [agent reference](/reference/agent#vision)
-lists the modes and restrictions.
-
-To check what a goal left on screen, follow it with an
-[assertion](/assertions) or a [locator check](/locators).
-
-## Tests without a screen
-
-A test that takes only `app` calls no model and drives no screen. Use it to
-check your API with `fetch` in the same run as UI tests. See
-[Testing APIs](/api-testing).
-
-## Structure
-
-```ts
-import { beforeEach, describe, test } from 'e2e';
-
-describe('billing', { tags: ['billing'] }, () => {
-  beforeEach(async ({ app }) => { await app.open('/billing'); });
-
-  test('upgrades', { retries: 2, timeout: 60_000 }, async ({ agent }) => {
-    await agent.act('upgrade to Pro');
-  });
-});
-
-test.skip('not ready yet', async () => {});
-test.only('just this one while I work', async () => {}); // rejected in CI
-
-test('needs two organizations', async ({ app }) => {
-  test.skip((await countOrganizations()) < 2, 'one organization: nothing to switch to');
-  await app.open('/organizations');
-});
-```
-
-`test.skip(condition, reason)` inside a body throws when the condition holds:
-the test stops at that line and is reported `skipped`, as in Playwright. Call
-it before the first step when it guards a precondition.
-
-`describe` and the hooks are also `test.describe`, `test.beforeEach`, and so
-on. A `beforeEach` imported from `e2e` sees the core fixtures; import
-`test`, `describe`, and the hooks from `@e2e-dev/web` (or `@e2e-dev/mobile`)
-to have `browser` (or `device`) typed from one import, and use
-`test.beforeEach` on a `test` you extended with your own fixtures.
-
-Common options include `timeout`, `retries`, and `tags`. Use `session` for
-state from a setup test, `agentContext` for extra instructions, and `agent`
-to choose a configured [agent](/agents). `trace` and `video` record one test
-without recording the suite; see [Watch it happen](/debugging#watch-it-happen).
-Group options apply to their tests. The [test reference](/reference/test)
-lists all options and defaults.
-
-Each test runs in its own attempt. Prepare the app data it needs through
-fixtures or hooks. When a flow must span several tests, mark the group
-`{ serial: true }`. Members share app state, run in order on one worker,
-and retry as a whole.
-
-Share setup and cleanup through `test.extend`. A fixture can create a test
-workspace before the test and delete it afterward, including when the test
-fails. See [Your own fixtures](/reference/test#your-own-fixtures).
-
-## Run one test
-
-<CodeGroup>
-```bash npm
-npx e2e run tests/signup.e2e.ts:12   # the test declared at line 12
-npx e2e run --grep checkout          # tests whose title matches
-npx e2e run --target web             # one target, by name
-```
-
-```bash pnpm
-pnpm exec e2e run tests/signup.e2e.ts:12   # the test declared at line 12
-pnpm exec e2e run --grep checkout          # tests whose title matches
-pnpm exec e2e run --target web             # one target, by name
-```
-
-```bash bun
-bunx e2e run tests/signup.e2e.ts:12   # the test declared at line 12
-bunx e2e run --grep checkout          # tests whose title matches
-bunx e2e run --target web             # one target, by name
-```
-</CodeGroup>
-
-The [CLI reference](/reference/cli#e2e-run) lists every flag.
-
-## Next
-
-<CardGroup cols={2}>
-  <Card title="Assertions" icon="circle-check" href="/assertions">
-    Check the screen with `assert`, `waitFor`, and `extract`.
-  </Card>
-  <Card title="Locators" icon="crosshairs" href="/locators">
-    Exact actions and checks with `screen` and `expect`.
-  </Card>
-</CardGroup>

package/docs/locators.mdx

@@ -1,64 +0,0 @@
----
-title: Locators
-description: Deterministic actions and checks with screen and expect.
----
-
-Use locators to interact with known controls and check exact values. Locator
-steps call no model.
-
-```ts title="tests/todos.e2e.ts"
-import { test, expect } from 'e2e';
-
-test('adds a todo', async ({ app, screen }) => {
-  await app.open('/todos');
-  await screen.getByLabel('New todo').fill('Buy milk');
-  await screen.getByRole('button', 'Add').tap();
-  await expect(screen.getByTestId('todo')).toHaveCount(1);
-});
-```
-
-An `expect` after an [`agent.act` goal](/goals) checks the exact result. It is
-also what lets the [replay cache](/cache) record the step and keep it.
-
-## Queries, actions, and assertions
-
-```ts
-// Semantic queries: role and name first, label second, test ID last.
-await screen.getByRole('button', 'Sign in').tap();
-await screen.getByLabel('Email').fill('ada@example.test');
-await screen.getByTestId('todo').first().tap();
-
-// Scope into a row, gesture like a user:
-const row = screen.getByRole('listitem').filter({ hasText: 'Design review' });
-await row.getByRole('button', 'Archive').tap();
-await screen.swipe({ direction: 'up', momentum: 'fast' });
-
-// Coordinates, for what the tree does not list:
-await screen.getByRole('image', 'Seat map').tap({ position: { x: 120, y: 48 } });
-await screen.swipe({ from: { x: 300, y: 400 }, to: { x: 40, y: 400 } });
-
-// Polling assertions:
-await expect(screen.getByRole('status')).toHaveText('2 remaining');
-await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible();
-```
-
-Actions and single-node reads or assertions fail when a query matches more
-than one node. Narrow it with `.filter()`, `.first()`, or `.nth()`.
-Locator assertions retry until they pass or time out. Direct reads such as
-`textContent()` do not retry a mismatched value.
-
-Use the `browser` fixture for browser features such as routes, cookies, dialogs,
-frames, and downloads; see [Working with the browser](/browser). The
-[screen reference](/reference/screen) lists every query and action, and the
-[expect reference](/reference/expect#locator-matchers) lists every matcher.
-
-## Next
-
-<CardGroup cols={2}>
-  <Card title="Signing in" icon="key" href="/authentication">
-    Sessions and secrets.
-  </Card>
-  <Card title="Agents and personas" icon="users" href="/agents">
-    Several agents in one suite, and running a flow as each of them.
-  </Card>
-</CardGroup>

package/docs/physical-devices.mdx

@@ -1,76 +0,0 @@
----
-title: Physical devices
-description: Run mobile tests on an iPhone or Android phone connected to your machine.
----
-
-agent-device drives a phone plugged into your machine the same way it
-drives a simulator, so a physical device needs no other engine or option.
-List what agent-device sees, then name the phone in `device` as it is
-listed:
-
-<CodeGroup>
-```bash npm
-npx agent-device devices --platform ios
-```
-
-```bash pnpm
-pnpm exec agent-device devices --platform ios
-```
-
-```bash bun
-bunx agent-device devices --platform ios
-```
-</CodeGroup>
-
-```ts
-mobile({ platform: 'ios', device: 'QA iPhone' })
-```
-
-Name a phone by its name, not its id. The engine reads a `device` string as
-an id only when it is a simulator UDID or an `emulator-<port>` serial, so a
-phone's UDID or adb serial is looked up as a name and not found. The first
-device with a matching name wins, so give each phone a unique name: two
-phones both called `iPhone` or `Pixel 8` would bind one phone to two
-workers. Without `device`, a connected phone counts as booted and joins the
-pool beside any booted simulators, so name it to be sure a run lands on it.
-
-Set the phone up once:
-
-- **iOS**: the phone is paired and trusted, unlocked, with Developer Mode
-  on. agent-device builds and signs its XCTest runner onto the phone, so set
-  `AGENT_DEVICE_IOS_TEAM_ID` to your team id and `AGENT_DEVICE_IOS_BUNDLE_ID`
-  to a unique reverse-DNS id (`com.example.agentdevice.runner`); the
-  signing profile must allow that id and `<id>.uitests`. Set them before the
-  first agent-device command, or run `npx agent-device daemon stop` after:
-  a running daemon keeps the environment it started with. The first run
-  builds the runner and takes longer. `app.appPath` must be a signed device
-  build: a simulator `.app` does not install on a phone.
-- **Android**: USB debugging is on and the phone is authorized, listed as
-  `device` in `adb devices`. `app.appPath` is the same `.apk`.
-
-`npx agent-device doctor` checks the setup, and
-`npx agent-device help physical-device` covers iOS signing in detail.
-
-Some device controls need a simulator or emulator, and fail on a phone with
-agent-device's message:
-
-- On an iPhone, every device setting is simulator-only: `app.permissions`
-  and `device.setPermission`, `app.clearState()`, `device.setNetwork`,
-  `device.setAirplaneMode`, `device.setLocation`, `device.clearLocation`,
-  `device.setAppearance`, the biometric methods, and `device.clearKeychain`.
-  So are `device.clipboard`, `device.setClipboard`, and `device.fold`. For a
-  fresh start, reinstall with `device.installApp(path, { reinstall: true })`.
-- An older iPhone that `xcrun devicectl list devices` does not list is
-  driven without CoreDevice: taps, typing, and the screen work, but
-  installs, `--video`, `device.openLink`, and `app.launchArguments` fail.
-  Install the build from Xcode and pin `app.bundleId`.
-- On an Android phone, `device.setLocation` is emulator-only, and
-  `device.setBiometrics('fingerprint', ...)` works only where the phone
-  exposes `cmd fingerprint`.
-- `--video` on an iPhone is built from screenshots at about 15 frames a
-  second, and drops frames.
-- A phone is not this machine, so `localhost` in the app is the phone
-  itself. Point the app at your machine's network address, or on Android
-  forward the port with `adb reverse tcp:<port> tcp:<port>`.
-
-To run on phones in a device farm, see [Hosted devices](/mobile#hosted-devices).

package/docs/viewports.mdx

@@ -1,146 +0,0 @@
----
-title: Testing viewports
-description: Run a suite at several sizes, resize inside a test, and know what the agent sees at each.
----
-
-Set a viewport on the target to run a suite at that size. Use
-`browser.setViewport()` to resize during a test. The agent uses the current size
-and cannot resize it.
-
-## One size per target
-
-`viewport` sets the initial size of each attempt, in pixels. The default is
-1280 by 720. Add one target for each size you want to test:
-
-```ts title="e2e.config.ts"
-import type { E2EConfig } from 'e2e';
-import { web } from '@e2e-dev/web';
-
-const app = { url: 'http://localhost:3000' };
-
-export default {
-  targets: [
-    { name: 'desktop', engine: web(), app },
-    { name: 'tablet', engine: web({ viewport: { width: 820, height: 1180 } }), app },
-    { name: 'phone', engine: web({ browser: 'webkit', viewport: { width: 390, height: 844 } }), app },
-  ],
-} satisfies E2EConfig;
-```
-
-Each test runs once per target, with a separate result labeled `desktop`,
-`tablet`, or `phone`. Run one size with `npx e2e run --target phone`, or list
-the selected test-target pairs with `npx e2e list`.
-
-To have the runner start one dev server for every size, give every target
-the same `app.command`: targets declaring one command share one process; see
-[several targets](/starting-your-app#several-targets). Every retry starts at
-the target's configured size.
-
-## Fill the window
-
-`viewport: null` sets no size, so the page fills the browser window. The
-agent, screenshots, and videos use that size. Use it in a headed run, or
-with a hosted browser whose live view would show a fixed 1280 by 720 page
-in one corner:
-
-```ts title="e2e.config.ts"
-{ name: 'window', engine: web({ viewport: null }), app }
-```
-
-The page takes the window's size, minus the toolbar in a headed browser. A
-local headed launch opens a small window; a hosted browser uses the window
-size you requested from the service. `browser.setViewport` still works.
-
-## Resize inside a test
-
-`browser.setViewport` resizes the page for the rest of the attempt, through
-`app.restart()` and `app.clearState()`:
-
-```ts
-import { test } from '@e2e-dev/web';
-import { expect } from 'e2e';
-
-test('the menu collapses on a narrow screen', async ({ app, screen, browser }) => {
-  await app.open('/');
-  await expect(screen.getByRole('navigation')).toBeVisible();
-
-  await browser.setViewport({ width: 390, height: 844 });
-  await expect(screen.getByRole('button', 'Menu')).toBeVisible();
-  await expect(screen.getByRole('navigation')).toBeHidden();
-});
-```
-
-Use this when one test needs to check a layout before and after resizing.
-For a whole suite, configure separate targets instead.
-
-Call it before `app.open` to load the first page at that size, so the app
-never renders its desktop layout first:
-
-```ts
-await browser.setViewport({ width: 390, height: 844 });
-await app.open('/');
-```
-
-The resize appears as a step in the report. To check the exact width, read
-it from the page:
-
-```ts
-const width = await browser.evaluate(() => window.innerWidth);
-expect(width).toBe(390);
-```
-
-## What the agent sees
-
-The agent receives the current viewport size with each observation.
-Screenshots cover that viewport. Off-screen elements can still appear in
-the tree, and tapping them scrolls them into view. The agent can scroll to
-reveal lazy-loaded content.
-
-Resize before calling `agent.act`:
-
-```ts
-test('the mobile menu reaches pricing', async ({ app, agent, browser }) => {
-  await browser.setViewport({ width: 390, height: 844 });
-  await app.open('/');
-  await agent.act('open the navigation menu and go to Pricing');
-  await expect(browser).toHaveURL('/pricing');
-});
-```
-
-## Scoping a test to one size
-
-`platforms` and `requires` cannot select a viewport; all Playwright targets
-use platform `web`. To skip a test at some sizes, check the width in the
-test:
-
-```ts
-test('the sidebar stays open', async ({ app, screen, browser }) => {
-  await app.open('/');
-  test.skip((await browser.evaluate(() => window.innerWidth)) < 768, 'desktop layout only');
-  await expect(screen.getByRole('complementary')).toBeVisible();
-});
-```
-
-To keep a whole group of size-specific tests apart, give them their own
-config and run it with `--config`.
-
-## Limits
-
-A narrow viewport tests responsive layout. It still uses a desktop browser
-with a scale factor of 1, desktop pointer behavior, and its usual user agent.
-It does not emulate touch, color scheme, or reduced-motion preferences.
-Use the [device engine](/mobile) for tests on a simulator or emulator.
-
-A video keeps the size the page had when it started recording; set
-`web({ screencast: { size } })` for one fixed size. A browser provider that
-records, such as [Kernel](/integrations/kernel#recordings), records the whole
-screen instead.
-
-<CardGroup cols={2}>
-  <Card title="Working with the browser" icon="globe" href="/browser">
-    Routes, cookies, dialogs, downloads, and the rest of the `browser` fixture.
-  </Card>
-  <Card title="Web engine reference" icon="masks-theater" href="/reference/web">
-    Every engine option and `browser` method, with timeouts and errors.
-  </Card>
-</CardGroup>