e2e 0.15.0 → 0.15.2

package/docs/agents.mdx

@@ -0,0 +1,217 @@
+---
+title: Agents and personas
+description: Configure several agents in one suite, pin tests to them, and run one flow as each persona.
+---
+
+Define named agents in `agents` to give tests different models, instructions,
+or tools. Tests use `default` unless you select another agent.
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { web } from '@e2e-dev/web';
+import { gateway } from 'ai';
+
+const vocabulary = 'Plans are called tiers. Billing lives under Settings. "Approve" on an order refunds it.';
+
+export default {
+  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
+  agents: {
+    default: { model: gateway('openai/gpt-6-luna-fast'), context: vocabulary },
+    thorough: {
+      model: gateway('anthropic/claude-opus-5'),
+      system: 'Verify every claim on screen before concluding a step.',
+      context: vocabulary,
+    },
+    buyer: { model: gateway('openai/gpt-6-luna-fast'), system: 'You are a first-time buyer.', context: vocabulary },
+    admin: { model: gateway('openai/gpt-6-luna-fast'), system: 'You manage orders and approve refunds.', context: vocabulary },
+  },
+} satisfies E2EConfig;
+```
+
+Each entry is one plain object. For the built-in agent it holds the model,
+the `system` prompt, the app's `context`, [tools](/tools), budgets, and
+`providerOptions`. For a [custom executor](/executors), put the executor under
+`executor`; the model, judge, context, and budgets still apply, while `system`
+and `tools` are rejected because the executor brings its own prompt and tools.
+
+Agents do not inherit from each other. Every entry starts from the built-in
+defaults, never from `default`'s values, so give each entry the model and
+context it needs, for example from a shared constant as above.
+
+`model` handles acting and judgments unless you set a separate `judge`.
+Models that receive screenshots must accept images. See [Models](/models) and
+the [config reference](/reference/config#agents).
+
+## Instructions and vocabulary
+
+An agent takes two kinds of text, and they go to different readers.
+
+| | `system` | `context` |
+|---|---|---|
+| Answers | How should the acting agent work? | What does this app call things? |
+| Who reads it | The act loop only | Every model call: act turns and the judges behind `assert`, `waitFor`, and `extract` |
+| Belongs here | Persona, caution, what to verify before finishing, how to treat unexpected dialogs | Names of screens and menus, where a feature lives, which button submits a form, domain terms |
+| Set on | `agents.<name>.system` | `agents.<name>.context`, at most 16 KiB |
+
+The split is deliberate. A judge decides whether the screen shows what the
+test claims, so it must know the app's words: a judge that does not know
+"plans are called tiers" cannot check that a tier was chosen. A judge must
+not know the actor's instructions, or a `system` prompt that says "consider a
+step done once the form closes" would lower the bar for the verdict too.
+Nothing in `system` ever reaches a judge.
+
+A rule of thumb: if the sentence would still be true for a different tester
+using the same app, it is `context`. If it would still be true for the same
+tester on a different app, it is `system`.
+
+```ts title="e2e.config.ts"
+agents: {
+  default: {
+    model: gateway('openai/gpt-6-luna-fast'),
+    // How to work: read by the act loop only.
+    system: 'Verify a visible confirmation before completing a step. Close cookie and feedback notices you did not ask for.',
+    // What things are called: read by every model call, judges included.
+    context: 'Plans are called tiers. Billing lives under Settings. Sales documents are quotes until "Convert to order".',
+    maxSteps: 40,
+  },
+},
+```
+
+`maxSteps` caps the actions one step may take and `maxModelCalls` its model
+calls; both default to 25 and go up to 100.
+
+A test or group can add vocabulary of its own with `agentContext`, which is
+appended after the agent's `context` for that test: the ids of seeded data,
+a feature flag that changes a label, anything only that test knows.
+
+```ts title="tests/billing.e2e.ts"
+import { describe, test } from 'e2e';
+
+describe('billing', { agentContext: 'The seeded workspace is called "Acme Trial".' }, () => {
+  test('upgrades the tier', async ({ agent }) => { /* ... */ });
+});
+```
+
+Keep instructions out of `agentContext` for the same reason they stay out of
+`context`: it reaches the judges.
+
+## Make the agent yours
+
+When a step fails or takes too many calls, inspect its report or run with
+`--debug` to see the transcript. Start with the instruction, then adjust the
+agent if the problem affects several tests:
+
+1. Use the labels on screen in the goal.
+2. Put shared vocabulary in `context`, or in `agentContext` for one test.
+3. Put instructions for the actor, such as verifying a visible confirmation
+   before completing a step, in `system`.
+4. Add [tools](/tools) for operations such as seeding a cart through a test API.
+5. Compare models or `providerOptions` by running the same tests with
+   `--agent <name>`.
+
+Named agents can use the [replay cache](/cache). Custom tools and executors
+have their own replay constraints, described under [project tools](/tools)
+and [custom executors](/executors#replay-cache).
+
+## Pick an agent per run
+
+Use `--agent` to choose the agent for tests that do not specify one:
+
+<CodeGroup>
+```bash npm
+npx e2e run --agent thorough tests/checkout.e2e.ts
+```
+
+```bash pnpm
+pnpm exec e2e run --agent thorough tests/checkout.e2e.ts
+```
+
+```bash bun
+bunx e2e run --agent thorough tests/checkout.e2e.ts
+```
+</CodeGroup>
+
+Several names run each unpinned test once per agent:
+
+<CodeGroup>
+```bash npm
+npx e2e run --agent buyer,admin
+```
+
+```bash pnpm
+pnpm exec e2e run --agent buyer,admin
+```
+
+```bash bun
+bunx e2e run --agent buyer,admin
+```
+</CodeGroup>
+
+## Pin an agent in the suite
+
+A test, describe block, or individual call can select an agent. The closest
+setting wins, in this order: call, test, enclosing groups, run.
+
+The abbreviated tests below show how to select an agent at each level.
+
+```ts title="tests/checkout.e2e.ts"
+import { describe, test } from 'e2e';
+
+describe('checkout as a first-time buyer', { agent: 'buyer' }, () => {
+  test('pays for one item', async ({ app, agent }) => { /* ... */ });
+
+  describe('refunds', { agent: 'admin' }, () => {
+    test('refunds the order', async ({ agent }) => { /* ... */ });
+  });
+
+  test('reorders, and the admin approves', async ({ app, agent }) => {
+    await app.open('/orders');
+    await agent.act('reorder the most recent order');
+    // The admin's act call sees what the earlier steps of this test did.
+    await agent.act('approve the pending reorder', { agent: 'admin' });
+    await agent.assert('the reorder shows as approved');
+  });
+});
+```
+
+`--agent` does not replace an explicitly selected agent. An unknown agent
+on a test or group fails during collection. An unknown agent on a call raises
+`INVALID_ARGUMENT`. Reports record the agent for each step.
+
+## One flow, several personas
+
+Select several agents to run the same test once with each:
+
+```ts title="tests/checkout.e2e.ts"
+import { describe, test } from 'e2e';
+
+describe('checkout', { agent: ['buyer', 'admin'] }, () => {
+  test('pays for one item', async ({ app, agent }) => { /* runs once per persona */ });
+  test('applies a promo code', async ({ app, agent }) => { /* ... */ });
+});
+```
+
+Each agent gets a separate result and artifact directory. Terminal and JUnit
+test names include the agent, such as `[admin]`.
+
+For the list above, `--agent admin` selects only `admin`. A flag with no
+matching name, such as `--agent thorough`, leaves both selected agents in place.
+
+A serial group runs as one unit per agent, so its members share the group's
+pin. A setup test pins at most one agent.
+
+Selecting an agent does not switch the signed-in user. For signed-in personas,
+pair each agent with a `session` in a separate describe block.
+[Signing in](/authentication#several-signed-in-users) shows how to create the
+sessions and run the tests.
+
+## Next
+
+<CardGroup cols={2}>
+  <Card title="Project tools" icon="wrench" href="/tools">
+    Give the agent a test API or a device toolkit.
+  </Card>
+  <Card title="Custom executors" icon="gears" href="/executors">
+    Customize how the agent chooses and performs actions.
+  </Card>
+</CardGroup>

package/docs/api-testing.mdx

@@ -0,0 +1,65 @@
+---
+title: Testing APIs
+description: Check your API with fetch and value matchers, in the same files and run as UI tests.
+---
+
+API tests run in the same files and the same run as UI tests. A test that
+takes only `app` calls no model. Call the API with `fetch` and check the
+response with the value matchers.
+
+```ts title="tests/users-api.e2e.ts"
+import { test, expect } from 'e2e';
+import { z } from 'zod';
+
+const User = z.object({ id: z.number(), name: z.string() });
+
+test('GET /api/users returns the seeded users', async ({ app }) => {
+  const response = await fetch(new URL('/api/users', app.baseUrl));
+  expect(response.status).toBe(200);
+  expect(response.headers.get('content-type')).toContain('application/json');
+  const users = expect(await response.json()).toMatchSchema(z.array(User));
+  expect(users).toHaveLength(3);
+  expect(users.map((user) => user.name)).toEqual(['Ada', 'Grace', expect.any(String)]);
+});
+```
+
+`toMatchSchema` takes any [Standard Schema](/reference/expect#matching-a-schema)
+(Zod, Valibot, ArkType), so the schema your app parses the API with checks
+that the API still matches it, and it returns the value typed.
+
+## Where the API runs
+
+`app.baseUrl` is the target's `app.url`, with the port the run allocated when
+it declared port 0, so the same file runs against the `app.command` the config
+starts and against a deployed preview. For state the API writes after it
+answers, [`expect.poll`](/reference/expect#polling-a-value) re-reads until a
+matcher passes.
+
+## Signed-in requests
+
+To call the API as a signed-in user, write [a fixture](/reference/test#your-own-fixtures)
+that sends `browser.cookies()` with each request. Every test registered through
+this `test` acquires `browser`, so it needs a web target:
+
+```ts title="tests/api.ts"
+import { test as base } from '@e2e-dev/web';
+
+export const test = base.extend<{ api: (path: string, init?: RequestInit) => Promise<Response> }>({
+  api: async ({ app, browser }, use) => {
+    await use(async (path, init) => {
+      const url = new URL(path, app.baseUrl);
+      if (url.origin !== new URL('/', app.baseUrl).origin) throw new Error(`${url} is not on the app`);
+      const headers = new Headers(init?.headers);
+      headers.set('cookie', (await browser.cookies()).map((c) => `${c.name}=${c.value}`).join('; '));
+      return fetch(url, { ...init, headers });
+    });
+  },
+});
+```
+
+## Without a browser
+
+On a web target, an API test still opens a browser context. A target
+without an engine opens nothing, but it has no `app.baseUrl`, so read the
+URL from an environment variable. `e2e init` sets up such a target when you
+choose **None** at the engine prompt.

package/docs/authentication.mdx

@@ -0,0 +1,345 @@
+---
+title: Signing in
+description: Sign in once, reuse the session across tests, and pass secrets without exposing them to the model.
+---
+
+Sign in once in a setup test and save a named session. Other tests restore
+that session to start signed in.
+
+<Tabs>
+  <Tab title="tests/auth.setup.e2e.ts">
+
+```ts
+import { test } from '@e2e-dev/web';
+import { expect, credentials } from 'e2e';
+
+test.setup('authenticate as admin', { sessions: ['admin'] }, async ({ app, screen, session, browser }) => {
+  const admin = credentials.user('admin');
+
+  await app.open('/login');
+  await screen.getByLabel('Username').fill(admin.username);
+  await screen.getByLabel('Password').fill(admin.password);
+  await screen.getByRole('button', 'Sign in').tap();
+
+  await expect(browser).toHaveURL('/dashboard');
+  await expect(screen.getByRole('status', 'Greeting')).toContainText('admin');
+
+  await session.save('admin');
+});
+```
+
+  </Tab>
+  <Tab title="tests/dashboard.e2e.ts">
+
+```ts
+import { test } from '@e2e-dev/web';
+import { expect } from 'e2e';
+
+test('the dashboard opens directly', { session: 'admin' }, async ({ app, screen, browser }) => {
+  await app.open('/dashboard');
+
+  await expect(browser).toHaveURL('/dashboard');
+  await expect(screen.getByRole('status', 'Greeting')).toHaveText(
+    'Welcome back, admin!',
+  );
+});
+```
+
+  </Tab>
+</Tabs>
+
+The runner runs the required setup even when you select only
+`tests/dashboard.e2e.ts`.
+
+## Sessions
+
+| Rule | If you break it |
+| --- | --- |
+| Setup tests are top-level, never inside `describe` | collection error |
+| Exactly one setup declares a given session name | collection error |
+| Every name in `sessions` is saved by the body | `SESSION_CONTRACT` failure |
+| Only declared names are saved | `SESSION_CONTRACT` failure |
+| The target's engine saves and restores state | collection error, before any test runs; a hosted browser leased per attempt has none |
+
+Setup tests run before any ordinary test for the same target. If a setup
+fails, its dependents are skipped with cause `setup-failed` and unrelated
+tests keep going.
+
+<Note>
+`session.save()` saves the current state without checking whether sign-in
+succeeded. Assert the post-login URL or a signed-in element before saving.
+</Note>
+
+On a browser target, a session holds cookies, local storage, and IndexedDB.
+It does not include server or database state. Restoring replaces the client
+state and leaves no page open, so each test calls `app.open()` first.
+
+Sessions last for one run. They are encrypted on disk with a key that lives
+only in runner memory, and deleted during cleanup. A session file is a bearer
+credential, so it is never reused across runs.
+
+## Credentials
+
+Declare credentials in config and read passwords from the environment:
+
+```ts title="e2e.config.ts"
+export default {
+  targets: [{ engine: web(), app: { url: 'http://127.0.0.1:3000' } }],
+  credentials: {
+    admin: {
+      username: 'admin@example.test',
+      password: process.env.ADMIN_PASSWORD ?? '',
+    },
+  },
+} satisfies E2EConfig;
+```
+
+Either field can also be overridden per run. The variable is the credential
+name uppercased, with every non-alphanumeric character replaced by `_`:
+
+<CodeGroup>
+```bash npm
+E2E_USER_ADMIN_USERNAME=admin E2E_USER_ADMIN_PASSWORD=... npx e2e run
+```
+
+```bash pnpm
+E2E_USER_ADMIN_USERNAME=admin E2E_USER_ADMIN_PASSWORD=... pnpm exec e2e run
+```
+
+```bash bun
+E2E_USER_ADMIN_USERNAME=admin E2E_USER_ADMIN_PASSWORD=... bunx e2e run
+```
+</CodeGroup>
+
+A string is checked when the config loads: while `ADMIN_PASSWORD` is unset,
+every command, `e2e list` included, fails with `INVALID_CONFIG`.
+
+### Keep the value out of the environment
+
+A string from `process.env` sits in the environment of the runner and of
+every process it starts, where any code in the run can read it. `password`
+also accepts a function, called on every authorized fill and nowhere else,
+so the value can come straight from a vault, or be a freshly computed
+one-time code, and never be an environment variable (only the vault client's
+own access is configured in the environment):
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { web } from '@e2e-dev/web';
+import { vault } from './support/vault.ts';
+
+export default {
+  targets: [{ engine: web(), app: { url: 'http://127.0.0.1:3000' } }],
+  credentials: {
+    admin: {
+      username: 'admin@example.test',
+      password: () => vault.read('e2e/admin-password'),
+    },
+  },
+} satisfies E2EConfig;
+```
+
+The function runs in the runner, after the fill is authorized, and its value
+goes to the field and the redaction ledger only. A function's value is
+redacted only from its first fill on, so prefer a string for a value the app
+can show before a test types it. The same applies to a `secrets` entry.
+
+## The password is a handle
+
+```ts
+import { credentials } from 'e2e';
+
+const admin = credentials.user('admin');
+
+admin.name;     // 'admin'
+admin.username; // a plain string
+admin.password; // Secret: opaque, named 'admin.password', with no plaintext accessor
+```
+
+The two fields are declared alike, but only the password is a secret. A
+username is an identifier the app shows on screen, in the account menu and
+the audit log, so the agent reads it and tests assert on it; redacting it
+would hide exactly what a test checks. The password is never shown, so the
+runner keeps it away from test code and the model altogether.
+
+A `Secret` has no `.value`. Three sinks accept one: the two below in a
+test, and an engine option that declares it, as
+[`basicAuth`](/starting-your-app#protected-previews) does:
+
+<Tabs>
+  <Tab title="Deterministic">
+
+```ts
+await screen.getByLabel('Password').fill(admin.password);
+```
+
+  </Tab>
+  <Tab title="Agentic">
+
+```ts
+await agent.act('Sign in', {
+  params: { username: admin.username, password: admin.password },
+});
+```
+
+  </Tab>
+</Tabs>
+
+In `agent.act`, the model sees the secret's name and purpose. The runner
+authorizes the fill and supplies the value. Secret values are redacted from
+model input and reports. Reading a secure field is denied, so check that
+sign-in succeeded instead.
+
+After a secret fill, screenshots are withheld for the rest of the attempt,
+and in tests that restore the session it saved. See
+[screenshots after a secret fill](/security#screenshots-after-a-secret-fill)
+for the rule and how to sign in with screenshots kept.
+
+## Secrets that are not passwords
+
+Use `secrets` for API keys, webhook tokens, or other sensitive values a test
+needs to enter:
+
+```ts title="e2e.config.ts"
+export default {
+  targets: [{ engine: web(), app: { url: 'http://127.0.0.1:3000' } }],
+  secrets: {
+    'stripe-key': process.env.STRIPE_TEST_KEY ?? '',
+    'webhook-token': process.env.WEBHOOK_TOKEN ?? '',
+  },
+} satisfies E2EConfig;
+```
+
+`E2E_SECRET_<NAME>` overrides an entry per run, with the same uppercasing rule
+as credentials, so a CI job can supply the value without touching the config:
+
+<CodeGroup>
+```bash npm
+E2E_SECRET_STRIPE_KEY=sk_test_... npx e2e run
+```
+
+```bash pnpm
+E2E_SECRET_STRIPE_KEY=sk_test_... pnpm exec e2e run
+```
+
+```bash bun
+E2E_SECRET_STRIPE_KEY=sk_test_... bunx e2e run
+```
+</CodeGroup>
+
+Get a handle with `secrets.get()` and use it in `fill` or `agent.act`:
+
+<Tabs>
+  <Tab title="Deterministic">
+
+```ts
+import { secrets } from 'e2e';
+
+await screen.getByLabel('Secret key').fill(secrets.get('stripe-key'));
+```
+
+  </Tab>
+  <Tab title="Agentic">
+
+```ts
+import { secrets } from 'e2e';
+
+await agent.act('Connect Stripe with the given key', {
+  params: { apiKey: secrets.get('stripe-key') },
+});
+```
+
+  </Tab>
+</Tabs>
+
+When the agent fills a password, the runner requires a password field. A
+`secrets` entry can fill any editable input. An unconfigured name throws
+`SECRET_UNAVAILABLE`.
+
+### Where the value goes
+
+The model sees only the secret's name and purpose; the runner checks the
+field and fills the value. Known values become `<secret:name>` in
+observations and reports. The [security model](/security#secrets) has the
+full rules.
+
+<Warning>
+Redaction matches full values and their encoded forms. A transformed value,
+such as the last four characters of a key, can still appear. Keep secret
+entry in setup tests and assert the result of submitting it. See
+[where redaction stops](/security#where-redaction-stops).
+</Warning>
+
+Credentials and secrets are separate namespaces. `secrets.get()` returns
+`secrets` entries only; a credential's password is `credentials.user(name).password`,
+which the agent and the report know as `<name>.password`. A credential is an
+account, a username with the secrets that sign it in, while a secret is one
+value, so `credentials.admin` and `secrets.admin` can both exist.
+
+In `e2e.config.ts`, `secrets.get()` hands an engine option a secret it needs
+before any test runs, such as a [basic-auth password](/starting-your-app#protected-previews).
+The config load checks the name, and the value is redacted once the engine
+resolves it, together with the base64 `Authorization` credential it becomes.
+The protection is text only: screenshots and the model's pixels are kept, so
+a page that renders the password on screen is not masked in them.
+
+## Several signed-in users
+
+Save one session per user, then run the same tests with each session and
+[agent](/agents):
+
+<Tabs>
+  <Tab title="tests/personas.setup.e2e.ts">
+
+```ts
+import { test } from '@e2e-dev/web';
+import { expect, credentials } from 'e2e';
+
+const PERSONAS = ['buyer', 'admin'] as const;
+
+test.setup('sign in each persona', { sessions: [...PERSONAS] }, async ({ app, screen, session, browser }) => {
+  for (const name of PERSONAS) {
+    const user = credentials.user(name);
+    await app.open('/login');
+    await screen.getByLabel('Username').fill(user.username);
+    await screen.getByLabel('Password').fill(user.password);
+    await screen.getByRole('button', 'Sign in').tap();
+    await expect(browser).toHaveURL('/dashboard');
+    await session.save(name);
+    await app.open('/logout');
+  }
+});
+```
+
+  </Tab>
+  <Tab title="tests/checkout.e2e.ts">
+
+```ts
+import { describe, test } from '@e2e-dev/web';
+
+for (const persona of ['buyer', 'admin'] as const) {
+  describe(`checkout as ${persona}`, { agent: persona, session: persona, tags: [persona] }, () => {
+    test('pays for one item', async ({ app, agent }) => {
+      await app.open('/cart');
+      await agent.act('pay for the item in the cart');
+      await agent.assert('the order confirmation is shown');
+    });
+  });
+}
+```
+
+  </Tab>
+</Tabs>
+
+Each name refers to an `agents` entry, a `credentials` entry, and a saved
+session. When the personas do not need separate sessions,
+[run one flow as several agents](/agents#one-flow-several-personas) instead.
+
+<CardGroup cols={2}>
+  <Card title="test reference" icon="flask" href="/reference/test">
+    `test.setup`, `sessions`, and the `session` option.
+  </Card>
+  <Card title="Config reference" icon="sliders" href="/reference/config#credentials">
+    The `credentials` and `secrets` blocks.
+  </Card>
+</CardGroup>