e2e 0.16.0 → 0.18.0

package/docs/writing-tests.mdx

@@ -1,256 +0,0 @@
----
-title: Writing tests
-description: Goals, checks, extraction, and the deterministic APIs.
----
-
-Use agent calls for goals and questions about the screen. Use `screen` and
-`expect` when you know the exact action or value to check. A test can freely mix both.
-
-```ts title="tests/todos.e2e.ts"
-import { test, expect } from 'e2e';
-
-test('todos survive a filter round-trip', async ({ app, agent, screen }) => {
-  await app.open('/todos');
-
-  await agent.act('add todos "Buy milk" and "Walk the dog", complete the first');
-  await agent.assert('only one todo remains open');
-
-  await screen.getByRole('tab', 'Done').tap();
-  await expect(screen.getByTestId('todo')).toHaveCount(1);
-});
-```
-
-## Goals: `agent.act`
-
-Give `act` one goal. The agent chooses and performs the actions, then returns
-when the goal passes or throws when it fails.
-
-```ts
-await agent.act('complete checkout with the test card');
-await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
-```
-
-Pass values through `params` so the agent uses your test data. For passwords,
-use a `Secret`, this prevents the model from seeing your secrets. See [Signing in](/authentication).
-
-```ts
-import { credentials } from 'e2e';
-
-const member = credentials.user('member');
-await agent.act('Sign in', {
-  params: { username: member.username, password: member.password },
-});
-```
-
-<Tip>
-**Write instructions as you would say them out loud**
-
-Name what is visible and prefer the exact wording on screen. One goal per
-call. The flow inside a goal is the model's job; the order of goals is yours.
-</Tip>
-
-For a value that changes on every run, wrap it in `unique()` so the
-[replay cache](/cache#what-must-match) can still replay the step:
-
-```ts
-import { expect, unique } from 'e2e';
-
-const email = `ada+${Date.now()}@example.test`;
-await agent.act('sign up with email {email}', { params: { email: unique(email) } });
-await expect(screen.getByText(email)).toBeVisible();
-```
-
-If a goal fails, check the report and make the instruction more specific.
-Use the labels on screen. Put vocabulary shared by several goals in the
-agent's `context`; see [Agents and personas](/agents).
-
-A step can pass, fail, or be blocked by something such as missing credentials.
-[How agent steps work](/agent-steps#how-a-step-ends) explains the verdicts and
-exit codes.
-
-## Checks: `assert`, `waitFor`, `extract`
-
-`assert` checks a statement against the current screen. A false statement
-fails with `ASSERTION_FAILED`. If the screen does not provide enough evidence,
-it fails with `ASSERTION_INCONCLUSIVE`.
-
-The judging model sees the statement and the current screen. It does not see
-earlier steps or the acting agent's summaries.
-
-```ts
-await agent.assert('the dashboard shows a trial badge');
-```
-
-`assert` checks the screen now. Use `waitFor` when the condition may become true
-later. It polls the screen and only calls the model when the observation changes.
-
-```ts
-await agent.waitFor('the export finishes and a download link appears', {
-  interval: 500,
-  timeout: 120_000,
-});
-```
-
-`extract` reads structured data from the screen. Provide a
-[Standard Schema](https://standardschema.dev) validator, such as Zod, to define
-the result.
-
-```ts
-import { z } from 'zod';
-
-const data = await agent.extract('every todo title and how many remain', {
-  schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }),
-});
-expect(data.titles).toContain('Buy milk');
-```
-
-<Tip>
-For model-generated descriptions, check the relevant meaning rather than the
-entire sentence. Use a locator assertion when the exact text matters.
-</Tip>
-
-### Judging pixels
-
-By default, judgments read a redacted text snapshot of the screen. Add
-`vision: true` for visual checks, such as charts or layout. Use
-`vision: 'only'` to judge the screenshot alone.
-
-```ts
-await agent.assert('the chart trends upward', { vision: true });
-await agent.assert('the search form is not covered by a banner', { vision: 'only' });
-```
-
-`act` requests screenshots itself when it needs them, so it has no `vision`
-option. On supported engines, it can tap and type at points in the image:
-
-```ts
-await agent.act('enter the code 3141 on the drawn keypad and press OK');
-```
-
-Images add input tokens. Screenshots are withheld for the rest of an attempt
-after a secret is filled. The [agent reference](/reference/agent#vision)
-lists the modes and restrictions.
-
-## Deterministic APIs: `screen` and `expect`
-
-Use locators to interact with known controls and check exact values.
-An `expect` after `act` checks the exact result. It is also what lets the
-[replay cache](/cache) record the step and keep it.
-
-```ts
-// Semantic queries: role and name first, label second, test ID last.
-await screen.getByRole('button', 'Sign in').tap();
-await screen.getByLabel('Email').fill('ada@example.test');
-await screen.getByTestId('todo').first().tap();
-
-// Scope into a row, gesture like a user:
-const row = screen.getByRole('listitem').filter({ hasText: 'Design review' });
-await row.getByRole('button', 'Archive').tap();
-await screen.swipe({ direction: 'up', momentum: 'fast' });
-
-// Coordinates, for what the tree does not list:
-await screen.getByRole('image', 'Seat map').tap({ position: { x: 120, y: 48 } });
-await screen.swipe({ from: { x: 300, y: 400 }, to: { x: 40, y: 400 } });
-
-// Polling assertions:
-await expect(screen.getByRole('status')).toHaveText('2 remaining');
-await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible();
-```
-
-Actions and single-node reads or assertions fail when a query matches more
-than one node. Narrow it with `.filter()`, `.first()`, or `.nth()`.
-Locator assertions retry until they pass or time out. Direct reads such as
-`textContent()` do not retry a mismatched value.
-
-Use the `browser` fixture for browser features such as routes, cookies, dialogs,
-frames, and downloads; see [Working with the browser](/browser). The
-[screen reference](/reference/screen) lists every query and action.
-
-Share setup and cleanup through `test.extend`. A fixture can create a test
-workspace before the test and delete it afterward, including when the test
-fails. See [Your own fixtures](/reference/test#your-own-fixtures).
-
-## Tests without a screen
-
-A test that takes only `app` calls no model and drives no screen. Use it to
-check your API with `fetch` in the same run as UI tests. See
-[Testing APIs](/api-testing).
-
-## Structure
-
-```ts
-import { beforeEach, describe, test } from 'e2e';
-
-describe('billing', { tags: ['billing'] }, () => {
-  beforeEach(async ({ app }) => { await app.open('/billing'); });
-
-  test('upgrades', { retries: 2, timeout: 60_000 }, async ({ agent }) => {
-    await agent.act('upgrade to Pro');
-  });
-});
-
-test.skip('not ready yet', async () => {});
-test.only('just this one while I work', async () => {}); // rejected in CI
-
-test('needs two organizations', async ({ app }) => {
-  test.skip((await countOrganizations()) < 2, 'one organization: nothing to switch to');
-  await app.open('/organizations');
-});
-```
-
-`test.skip(condition, reason)` inside a body throws when the condition holds:
-the test stops at that line and is reported `skipped`, as in Playwright. Call
-it before the first step when it guards a precondition.
-
-`describe` and the hooks are also `test.describe`, `test.beforeEach`, and so
-on. A `beforeEach` imported from `e2e` sees the core fixtures; import
-`test`, `describe`, and the hooks from `@e2e-dev/web` (or `@e2e-dev/mobile`)
-to have `browser` (or `device`) typed from one import, and use
-`test.beforeEach` on a `test` you extended with your own fixtures.
-
-Common options include `timeout`, `retries`, and `tags`. Use `session` for
-state from a setup test, `agentContext` for extra instructions, and `agent`
-to choose a configured [agent](/agents). `trace` and `video` record one test
-without recording the suite; see [Watch it happen](/debugging#watch-it-happen).
-Group options apply to their tests. The [test reference](/reference/test)
-lists all options and defaults.
-
-Each test runs in its own attempt. Prepare the app data it needs through
-fixtures or hooks. When a flow must span several tests, mark the group
-`{ serial: true }`. Members share app state, run in order on one worker,
-and retry as a whole.
-
-## Run one test
-
-<CodeGroup>
-```bash npm
-npx e2e run tests/signup.e2e.ts:12   # the test declared at line 12
-npx e2e run --grep checkout          # tests whose title matches
-npx e2e run --target web             # one target, by name
-```
-
-```bash pnpm
-pnpm exec e2e run tests/signup.e2e.ts:12   # the test declared at line 12
-pnpm exec e2e run --grep checkout          # tests whose title matches
-pnpm exec e2e run --target web             # one target, by name
-```
-
-```bash bun
-bunx e2e run tests/signup.e2e.ts:12   # the test declared at line 12
-bunx e2e run --grep checkout          # tests whose title matches
-bunx e2e run --target web             # one target, by name
-```
-</CodeGroup>
-
-The [CLI reference](/reference/cli#e2e-run) lists every flag.
-
-## Next
-
-<CardGroup cols={2}>
-  <Card title="Signing in" icon="key" href="/authentication">
-    Sessions and secrets.
-  </Card>
-  <Card title="Agents and personas" icon="users" href="/agents">
-    Several agents in one suite, and running a flow as each of them.
-  </Card>
-</CardGroup>