e2e 0.15.1 → 0.16.0

package/docs/models.mdx

@@ -0,0 +1,241 @@
+---
+title: Models
+description: Configure a model with your own subscription, an API key, or a local server.
+---
+
+Agent steps use the model configured in `e2e.config.ts`. You can use an
+[existing subscription](/subscriptions), an API key, or a local server.
+`e2e init` offers these choices and writes the config for you. For the first
+setup, follow the [quickstart](/quickstart#use-a-subscription-or-api-key).
+
+Tests without agent steps do not need a model. For the built-in agent,
+choose a model that supports tool calls and images. Agent steps need the `ai`
+package installed, whichever provider you use:
+
+<CodeGroup>
+```bash npm
+npm install -D ai
+```
+
+```bash pnpm
+pnpm add -D ai
+```
+
+```bash bun
+bun add -d ai
+```
+</CodeGroup>
+
+There is no default model or shared API-key variable. The config selects the
+model, and the provider reads its own key from the environment, such as
+`AI_GATEWAY_API_KEY`. For a subscription, run `npx e2e login` instead.
+
+## Pick a gateway or a provider
+
+A gateway gives you access to several providers with one API key. A provider
+package connects to one provider directly. To change an existing config,
+replace its model import and `model` value:
+
+<Tabs>
+  <Tab title="Vercel AI Gateway">
+
+`gateway` is included in the `ai` package. Set `AI_GATEWAY_API_KEY`, or use
+a [Vercel OIDC token](#vercel-oidc-instead-of-a-key) without a key.
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { web } from '@e2e-dev/web';
+import { gateway } from 'ai';
+
+export default {
+  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
+  agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
+} satisfies E2EConfig;
+```
+  </Tab>
+  <Tab title="OpenRouter">
+
+Install `@openrouter/ai-sdk-provider` and set `OPENROUTER_API_KEY`.
+
+```ts title="Model setup"
+import { openrouter } from '@openrouter/ai-sdk-provider';
+
+const agent = { model: openrouter('openai/gpt-6-luna-fast') };
+```
+
+Use `agent` as `agents.default` in your config.
+  </Tab>
+  <Tab title="Provider API key">
+
+Install the provider's AI SDK package and set its API key. For OpenAI,
+install `@ai-sdk/openai` and set `OPENAI_API_KEY`:
+
+```ts title="Model setup"
+import { openai } from '@ai-sdk/openai';
+
+const agent = { model: openai('gpt-6-luna') };
+```
+
+Use `agent` as `agents.default`. Other AI SDK providers work the same way,
+with their own package, model IDs, and authentication settings.
+  </Tab>
+  <Tab title="Local or self-hosted">
+
+Install `@ai-sdk/openai-compatible`. Set the endpoint URL and a model ID
+served by that endpoint:
+
+```ts title="Model setup"
+import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
+
+const local = createOpenAICompatible({
+  name: 'local',
+  baseURL: 'http://127.0.0.1:11434/v1',
+  // apiKey: process.env.LLM_API_KEY,
+});
+
+const agent = { model: local.chatModel('your-model-id') };
+```
+
+Use `agent` as `agents.default`. Add `apiKey` if the server requires one.
+  </Tab>
+</Tabs>
+
+Model IDs depend on the provider. Gateways usually use `provider/model`;
+direct providers use their own model IDs.
+
+DeepSeek's V4 models (`@ai-sdk/deepseek`) think by default, and in that mode
+they refuse the forced tool choice agent steps ask for. The step is sent again
+with `auto` and an instruction to reply with tool calls only, and later steps
+on that model start that way, so thinking can stay on.
+
+### Vercel OIDC instead of a key
+
+Without `AI_GATEWAY_API_KEY`, `gateway()` authenticates with a Vercel OIDC
+token. Sign in to the Vercel CLI and link the project once; the gateway then
+mints a short-lived token for that project and refreshes it when it expires:
+
+<CodeGroup>
+```bash npm
+npx vercel login
+npx vercel link
+npx e2e run
+```
+
+```bash pnpm
+pnpm dlx vercel login
+pnpm dlx vercel link
+pnpm exec e2e run
+```
+
+```bash bun
+bunx vercel login
+bunx vercel link
+bunx e2e run
+```
+</CodeGroup>
+
+The token is a valid `VERCEL_OIDC_TOKEN`, otherwise one minted for the
+project in `.vercel/project.json`, found by walking up from the working
+directory. e2e loads no `.env` files, so a token pulled with
+`npx vercel env pull` reaches the gateway only when your shell exports it or
+the config loads the file with
+[`process.loadEnvFile()`](/reference/config#loading).
+
+An OIDC failure is the same `MODEL_PROVIDER_FAILED` as a bad key, and the AI
+SDK's message names only `AI_GATEWAY_API_KEY`. With no key set, check that the
+Vercel CLI is signed in and that `.vercel/project.json` exists.
+
+## Several models in one config
+
+Give agents different names to use different models:
+
+```ts title="e2e.config.ts"
+export default {
+  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
+  agents: {
+    default: { model: gateway('openai/gpt-6-luna-fast') },
+    careful: { model: gateway('anthropic/claude-sonnet-5') },
+  },
+} satisfies E2EConfig;
+```
+
+Run with `npx e2e run --agent careful`, or
+[choose an agent in a test](/agents). Each model can use a different provider.
+Agents do not inherit from each other: an entry without a `model` does not
+use `default`'s, see [Agents and personas](/agents).
+
+### Switch a model with an env var
+
+To switch a model without editing the config, read your own environment
+variable:
+
+```ts title="Agent setup"
+const agent = {
+  model: gateway(process.env.MODEL ?? 'openai/gpt-6-luna-fast'),
+};
+```
+
+### A judge of its own
+
+By default, the same model drives `agent.act` and answers `agent.assert`,
+`agent.waitFor`, and `agent.extract`. Set `judge` to use a separate model
+for those checks:
+
+```ts title="Agent setup"
+const agent = {
+  model: gateway('openai/gpt-6-luna-fast'),
+  judge: gateway('anthropic/claude-sonnet-5'),
+};
+```
+
+Judgments use the current screen and the question, without the acting
+agent's transcript or summaries. Reports include the model used for each
+judgment. See [Custom executors](/executors) if you replace the built-in agent.
+
+## Data retention and compliance
+
+Model input is the redacted screen, the step instruction, and the
+screenshots the agent asks for. e2e reads nothing back from the provider
+after a response, so a run needs no state kept on the provider's side.
+
+**OpenAI and Azure OpenAI.** Every request carries `store: false` and a
+prompt cache key. The provider keeps no copy of the conversation, a
+reasoning model's earlier turns travel inline as encrypted content, and the
+prompt cache still serves the repeated prefix. Organizations with zero data
+retention need no configuration. To keep stored responses, set `store` in
+`providerOptions`; the key is `azure` for Azure OpenAI:
+
+```ts title="Agent setup"
+const agent = {
+  model: openai('gpt-6-luna'),
+  providerOptions: { openai: { store: true } },
+};
+```
+
+**Anthropic under a BAA.** Agent steps use the Messages API with tool use,
+prompt caching, images, and structured outputs, all eligible under
+Anthropic's HIPAA readiness. The agreement covers `api.anthropic.com` only:
+call it directly with `@ai-sdk/anthropic` rather than through a gateway or
+OpenRouter, and leave `judge` unset so one agreement covers every request.
+Anthropic's Covered Models require 30-day retention and are not offered to
+organizations with zero data retention.
+
+## What fails, and where
+
+| Problem | Code | Fix |
+| --- | --- | --- |
+| A model is a string instead of an AI SDK instance | `INVALID_CONFIG` | Construct it, for example `gateway('openai/gpt-6-luna-fast')`. |
+| The built-in agent has no model | `MODEL_UNAVAILABLE` | Set `agents.<name>.model`. |
+| Authentication fails on the first model call | `MODEL_PROVIDER_FAILED` | Check the provider's API key or the linked Vercel project, or run `npx e2e login` for a subscription. |
+| The `ai` package is missing | `MODEL_UNAVAILABLE` | Run `npm install -D ai`. |
+| A step's second turn fails with `Item with id 'rs_…' not found` | `MODEL_PROVIDER_FAILED` | Leave `store` off in `providerOptions`, the default, or call OpenAI directly instead of through a gateway. |
+| A custom executor's own `model` or `judge` differs from the agents entry's | `INVALID_CONFIG` | See [Custom executors](/executors#keep-the-loop-change-the-vocabulary). |
+
+<CardGroup cols={2}>
+  <Card title="Subscriptions" icon="key" href="/subscriptions">
+    Sign in with ChatGPT, Copilot, or SuperGrok.
+  </Card>
+  <Card title="Agents and personas" icon="users" href="/agents">
+    Choose agents for tests and signed-in users.
+  </Card>
+</CardGroup>

package/docs/quickstart.mdx

@@ -0,0 +1,260 @@
+---
+title: Quickstart
+description: Set up e2e with one prompt to your coding agent, or step by step.
+---
+
+import { SetupPrompt } from './snippets/setup-prompt-card.jsx';
+
+export const setupPrompt = `Set up e2e end-to-end tests in this project. Docs: \`https://e2e.tester.army/docs/quickstart.md\`
+
+1. Run \`npx e2e init --yes\` (or the pnpm or bun equivalent for this project). It writes a web config using Vercel AI Gateway, an example test, the e2e skill, and MCP config. It installs nothing yet.
+2. Read the e2e skill (\`npx e2e guide\` prints it, \`npx e2e guide <topic>\` prints a topic) and follow it for every step below.
+3. Point the target at this app. For a web app, set its URL and the command that starts its dev server. For an iOS or Android app, replace @e2e-dev/web and playwright with @e2e-dev/mobile and rewrite the example test for the mobile engine.
+4. Ask me which model to use for agent steps: a ChatGPT, GitHub Copilot, or SuperGrok subscription, an API key, or a local model. Configure it with its provider package. If it needs a sign-in, give me the \`npx e2e login\` command and wait until I confirm.
+5. Install dependencies, run the example test, and fix setup errors until it passes.
+6. Write one test for the most important user flow in this app, run it, and iterate until it passes.`;
+
+## Set up with your coding agent
+
+Open Claude Code, Codex, Cursor, or another coding agent in your app's
+directory and paste this prompt:
+
+<SetupPrompt>
+
+<Prompt description={setupPrompt} children={setupPrompt} actions={["copy", "cursor"]} />
+
+</SetupPrompt>
+
+The agent installs the [e2e skill](/coding-agents), so later prompts like
+"add a test for checkout" or "why did this run fail" work without more setup.
+
+<Note>
+Use Node.js 22.12 or newer. On Windows, run inside WSL. Mobile tests also
+need Xcode with an iOS simulator runtime, or the Android SDK with an emulator.
+</Note>
+
+## Set up manually
+
+### Run your first test
+
+<Steps>
+  <Step title="Create the test project">
+  In your app's directory:
+
+  <CodeGroup>
+  ```bash npm
+  npx e2e init
+  ```
+
+  ```bash pnpm
+  pnpm dlx e2e init
+  ```
+
+  ```bash bun
+  bunx e2e init
+  ```
+  </CodeGroup>
+
+  Choose **Web (Playwright)** or **Mobile (iOS/Android)**, then a
+  subscription or model provider, or **None** for tests without AI. The
+  wizard writes a config and an example test, adds dependencies, and offers
+  to set up your coding agent. Existing files stay intact.
+
+  <Tabs>
+    <Tab title="Web">
+    Start your app's dev server at `http://localhost:3000`, or set its address
+    in `e2e.config.ts`. You can also [let e2e start it](/starting-your-app).
+
+    <Accordion title="Example web config and test">
+    This config uses Vercel AI Gateway. The wizard configures the provider you chose.
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { web } from '@e2e-dev/web';
+import { gateway } from 'ai';
+
+export default {
+  agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
+  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
+} satisfies E2EConfig;
+```
+
+```ts title="tests/example.e2e.ts"
+import { test } from '@e2e-dev/web';
+import { expect } from 'e2e';
+
+test('app opens', async ({ app, browser }) => {
+  await app.open('/');
+  await expect(browser.locator('body')).toBeVisible();
+});
+```
+    </Accordion>
+    </Tab>
+    <Tab title="Mobile">
+    Run `npx agent-device doctor` to check your setup. The example opens
+    Settings so you can try it before installing your own app. The wizard
+    chooses iOS on macOS and Android elsewhere.
+
+    <Accordion title="Example iOS config and test">
+    This config uses Vercel AI Gateway. For Android, the wizard sets
+    `platform: 'android'` and `app: { bundleId: 'com.android.settings' }`.
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { mobile } from '@e2e-dev/mobile';
+import { gateway } from 'ai';
+
+export default {
+  agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
+  targets: [{ name: 'ios', engine: mobile({ platform: 'ios' }), app: { bundleId: 'Settings' } }],
+  workers: 1,
+} satisfies E2EConfig;
+```
+
+```ts title="tests/example.e2e.ts"
+import { test } from '@e2e-dev/mobile';
+import { expect } from 'e2e';
+
+test('Settings opens', async ({ app, screen }) => {
+  await app.open();
+  await expect(screen.getByRole('button', 'General')).toBeVisible();
+});
+```
+    </Accordion>
+
+    See [Testing iOS and Android](/mobile) to use your own app.
+    </Tab>
+  </Tabs>
+  </Step>
+  <Step title="Run the example">
+  <CodeGroup>
+  ```bash npm
+  npx e2e run
+  ```
+
+  ```bash pnpm
+  pnpm exec e2e run
+  ```
+
+  ```bash bun
+  bunx e2e run
+  ```
+  </CodeGroup>
+
+  e2e downloads a browser or boots the simulator or emulator, checks that
+  the app opens, and writes `.e2e/report.json`. No model calls yet, so no
+  sign-in or API key needed.
+  </Step>
+</Steps>
+
+### Use a subscription or API key
+
+Connect the provider you chose in the wizard.
+
+<Tabs>
+  <Tab title="Subscription">
+  Sign in to your existing plan. No API key needed.
+
+  | Subscription | Command |
+  | --- | --- |
+  | ChatGPT Plus or Pro | `npx e2e login openai` |
+  | GitHub Copilot | `npx e2e login github-copilot` |
+  | SuperGrok or X Premium+ | `npx e2e login spacexai` |
+
+  Copilot reuses your GitHub CLI login if available. See
+  [Subscriptions](/subscriptions) for details.
+  </Tab>
+  <Tab title="API key">
+  Set your provider's key in the terminal where you run e2e. For Vercel AI
+  Gateway:
+
+  ```bash
+  export AI_GATEWAY_API_KEY="your-api-key"
+  ```
+
+  A project linked with `vercel link` can use a
+  [Vercel OIDC token](/models#vercel-oidc-instead-of-a-key) instead.
+  OpenRouter uses `OPENROUTER_API_KEY`. See [Models](/models) for other
+  providers and local models.
+  </Tab>
+</Tabs>
+
+### Add an agent step
+
+Give `agent.act` one goal per call, then check the result. Adapt the
+example to a flow in your app:
+
+<Tabs>
+  <Tab title="Web">
+```ts title="tests/agent.e2e.ts"
+import { test } from '@e2e-dev/web';
+import { expect } from 'e2e';
+
+test('a visitor signs up for a trial', async ({ app, agent, screen }) => {
+  await app.open('/');
+
+  await agent.act('sign up for a free trial as {name} with email {email}', {
+    params: { name: 'Ada Lovelace', email: 'ada@example.test' },
+  });
+
+  await agent.assert('the welcome screen greets Ada by name');
+  await expect(screen.getByRole('status')).toContainText('trial');
+});
+```
+  </Tab>
+  <Tab title="Mobile (iOS)">
+```ts title="tests/agent.e2e.ts"
+import { test } from '@e2e-dev/mobile';
+import { expect } from 'e2e';
+
+test('the agent opens General', async ({ agent, app, device }) => {
+  await app.open();
+  await agent.act('open {section} settings', { params: { section: 'General' } });
+
+  await agent.assert('the General settings screen is showing');
+  await expect(device.locator('role=NavigationBar id=General')).toBeVisible();
+});
+```
+
+  `app.open()` launches the configured app fresh; without it, a test starts
+  where the previous one left the device. `device.locator` uses
+  [agent-device selectors](/mobile#the-device-fixture).
+  </Tab>
+</Tabs>
+
+`params` fill the goal's placeholders. `agent.assert` judges the screen;
+`expect` checks an exact value.
+
+<CodeGroup>
+```bash npm
+npx e2e run tests/agent.e2e.ts
+```
+
+```bash pnpm
+pnpm exec e2e run tests/agent.e2e.ts
+```
+
+```bash bun
+bunx e2e run tests/agent.e2e.ts
+```
+</CodeGroup>
+
+Chose **None** in the wizard? [Add a model](/models) first. Add `--headed`
+to watch a web test, `--no-cache` to skip replaying cached actions.
+
+## Next
+
+<CardGroup cols={2}>
+  <Card title="Writing tests" icon="pen-to-square" href="/writing-tests">
+    Write goals, check outcomes, and extract data.
+  </Card>
+  <Card title="Signing in" icon="key" href="/authentication">
+    Reuse an app login and keep passwords out of model input.
+  </Card>
+  <Card title="Debugging" icon="bug" href="/debugging">
+    Read failures, screenshots, and traces.
+  </Card>
+  <Card title="Continuous integration" icon="code-branch" href="/ci">
+    Run your tests in CI.
+  </Card>
+</CardGroup>