e2e 0.15.1 → 0.15.2

package/docs/telemetry.mdx

@@ -0,0 +1,185 @@
+---
+title: Telemetry
+description: What the CLI sends, what it never sends, and how to turn it off.
+---
+
+The `e2e` CLI sends anonymous usage data so we can see which commands,
+engines, and features are used and where runs fail. It is on by default.
+Telemetry includes counts and version information, but no test content, app
+content, or credentials.
+
+## Opt out
+
+Any one of these turns it off:
+
+<CodeGroup>
+```bash npm
+npx e2e telemetry disable   # saved in ~/.config/e2e/telemetry.json
+export E2E_TELEMETRY_DISABLED=1          # for a shell or a CI job
+export DO_NOT_TRACK=1                    # the cross-tool convention
+```
+
+```bash pnpm
+pnpm exec e2e telemetry disable   # saved in ~/.config/e2e/telemetry.json
+export E2E_TELEMETRY_DISABLED=1          # for a shell or a CI job
+export DO_NOT_TRACK=1                    # the cross-tool convention
+```
+
+```bash bun
+bunx e2e telemetry disable   # saved in ~/.config/e2e/telemetry.json
+export E2E_TELEMETRY_DISABLED=1          # for a shell or a CI job
+export DO_NOT_TRACK=1                    # the cross-tool convention
+```
+</CodeGroup>
+
+`npx e2e telemetry` prints the status and, when it is off, why.
+`enable` reverses a saved `disable`. Environment variables take precedence.
+Running commands check the saved setting again before sending, so disabling
+telemetry also applies to commands already running in another terminal.
+
+## See what would be sent
+
+<CodeGroup>
+```bash npm
+E2E_TELEMETRY_DEBUG=1 npx e2e run
+```
+
+```bash pnpm
+E2E_TELEMETRY_DEBUG=1 pnpm exec e2e run
+```
+
+```bash bun
+E2E_TELEMETRY_DEBUG=1 bunx e2e run
+```
+</CodeGroup>
+
+This prints the exact event payloads to stderr as `[telemetry] {...}` and
+sends nothing.
+
+## What is sent
+
+Each command sends a session event. `run` and `explore` also send a run event,
+and `init` an init event. `--help` and `--version` send nothing, also when
+asked on a command. Events go in one request when the command ends, with a
+two-second deadline for all telemetry work.
+
+Every event carries these properties.
+
+Who ran it:
+
+| Property | Value |
+| --- | --- |
+| `distinct_id` | A random id stored in the preferences file. CI uses `ci:<vendor>` instead, and a platform that runs e2e for its users uses `fleet:<name>` when it sets `E2E_TELEMETRY_FLEET` |
+| `project_id` | A hash derived from the repository's root commit, shared across full clones. Outside git or in a shallow clone, a hash of the project path with a random local salt. `null` in CI when no root commit is available |
+| `session_id` | A random id for this one invocation |
+| `first_run`, `days_since_first_run` | Whether this command created the preferences file, and whole days since it was created. `null` in CI and in a fleet, where no machine persists, and for a preferences file from before the time was recorded |
+
+Where it ran:
+
+| Property | Value |
+| --- | --- |
+| `os`, `os_release`, `arch`, `cpu_count`, `memory_gb`, `docker`, `wsl`, `tty` | The machine class |
+| `sandbox` | The microVM the kernel announces itself as (`firecracker`), else `null` |
+| `ci`, `ci_name` | Whether `CI` is set, and the vendor when one announces itself (`github-actions`, `gitlab`, `circleci`, ...) |
+| `fleet` | The value of `E2E_TELEMETRY_FLEET` when it is a plain token, `other` when it is not, `null` when unset |
+| `coding_agent` | The coding agent whose shell ran the command, when one announces itself (`claude-code`, `cursor`, `codex`, ...) |
+
+Versions:
+
+| Property | Value |
+| --- | --- |
+| `e2e_version`, `package_manager` | The e2e version and the package manager that invoked the CLI |
+| `runtime`, `runtime_version`, `node_version` | `node`, `bun`, or `deno`, and its version; `node_version` is Node's own version, or the Node compatibility version under Bun or Deno |
+| `$lib`, `$lib_version`, `$process_person_profile`, `$geoip_disable` | `e2e`, its version, `false`, and `true`: PostHog keeps no person profile and derives no location |
+
+`e2e_cli_session`, once per command:
+
+| Property | Value |
+| --- | --- |
+| `command` | `init`, `run`, `list`, `cache ls`, `guide`, `telemetry`, ... |
+| `flags` | The names of the flags given, such as `--headed` or `--tag`. Never their values |
+| `exit_code`, `duration_ms` | How the command ended and how long it took |
+| `error_code` | The runner code of the failure that ended the command before it could run (`CONFIG_NOT_FOUND`, `INVALID_CONFIG`, ...), `CLI_USAGE` for a flag or argument the CLI rejected, `null` otherwise. A code that is not an upper-case token is reported as `OTHER` |
+
+`e2e_init_completed`, once per `e2e init`:
+
+| Property | Value |
+| --- | --- |
+| `result` | `scaffolded`, `already-initialized`, `cancelled`, `install-failed`, `not-interactive`, or `invalid-project` |
+| `yes`, `existing_config` | Whether `--yes` was given, and whether a config already existed so no engine or gateway was asked |
+| `engine`, `gateway` | The ids of the engine and the model gateway chosen for a new config (`web`, `mobile`, `none`; `vercel`, `openrouter`, `openai-compatible`, `chatgpt`, `copilot`, `grok`, `none`), `null` when not asked. Never the endpoint typed for an OpenAI-compatible gateway |
+| `skill`, `mcp`, `install` | Whether the agent skill, the MCP registration, and the dependency install were chosen |
+
+`e2e_run_completed`, once per `e2e run` or `e2e explore`, built from the numbers in `report.json`:
+
+| Property | Value |
+| --- | --- |
+| `status`, `exit_code`, `duration_ms` | The run's outcome |
+| `flags` | The names of the flags given to the run command, never their values |
+| `tests_discovered`, `tests_selected`, `tests_executed`, `tests_passed`, `tests_failed`, `tests_flaky`, `tests_skipped` | The summary counts |
+| `attempts_total`, `tests_retried` | How many attempts the run made across tests and serial groups, and how many of them needed more than one |
+| `targets`, `platforms`, `engines` | How many targets, on which platforms, and with which engines and versions, as your config declares them: `web`, `ios`, `web@0.6.1`, or your own engine's name and version. A name that is not a plain token of letters, digits, dots, dashes, and underscores is reported as `other` |
+| `steps_total`, `steps_agent`, `steps_locator`, `steps_assertion`, `steps_screen`, `steps_app`, `steps_session`, `steps_resource` | Step counts by kind |
+| `agent_steps_replayed`, `agent_steps_partial`, `agent_steps_missed`, `agent_steps_vision` | How the replay cache served the agent steps, and how many used pixel evidence |
+| `agent_actions` | How often the agent took each action, by the runner's own names: `{"tap": 12, "typeText": 3, "scroll": 2}`. A project's own tools are counted together as `tool` and anything else as `other`; no other name is sent |
+| `model_gateway`, `model_provider`, `model_id`, `model_calls`, `model_tokens`, `model_cached_tokens`, `estimated_cost_usd` | For the first model-backed step: the AI SDK provider that served it (`gateway` for the Vercel AI Gateway, `openrouter.chat`, `openai.chat`, ...), the vendor (the first segment of a `vendor/model` id, else the provider), and the public model id (a fine-tuned or routed id is reported as `other`); then the run's totals |
+| `cost_source` | `provider` when the provider priced the calls, `none` when the run made model calls nobody priced, `null` for a run without model calls |
+| `artifact_bytes` | Size of the artifacts the run wrote |
+| `errors`, `error_codes` | How many run-level errors, and the distinct error codes seen anywhere in the run (`LOCATOR_NOT_FOUND`, `APP_UNREACHABLE`, ...), at most 20. A code that is not an upper-case token, the shape of every runner code, is reported as `OTHER` |
+| `primary_error_code` | The first error the run recorded, which is the one that decided its status: a run-level error, else the failed attempt's verdict, else a step's; `null` for a clean run |
+| `error_kinds` | For `ENGINE_FAILURE`, `MODEL_PROVIDER_FAILED`, and `ERROR`, whose code says where a failure surfaced but not what it was: the code and a kind read off the message against a closed list (`auth`, `rate-limit`, `overloaded`, `timeout`, `network`, `server`, `bad-request`, `device`, `daemon`, `browser`, `permission`, `not-found`, `memory`, `other`), as `MODEL_PROVIDER_FAILED:rate-limit`. Only the kind's name leaves the machine, never the message |
+
+## What is never sent
+
+Test titles, file paths, URLs, origins, instructions, observations, page
+content, screenshots, error messages, stack traces, environment variable
+values, credentials, or file contents. Events include only the properties
+listed above.
+
+## In CI
+
+CI runs are attributed to the CI vendor rather than to a machine, no
+preferences file is written on the runner, and no notice is printed. Set
+`E2E_TELEMETRY_DISABLED: '1'` in the job's `env` to send nothing from CI.
+
+## In a fleet
+
+A platform that runs e2e on its users' behalf starts each run on a fresh
+machine, and every fresh machine would otherwise count as a new user. Set
+`E2E_TELEMETRY_FLEET=<name>` in those machines and they are attributed to
+`fleet:<name>` instead, like CI: no preferences file, no notice, and no
+project id outside git. Or set `E2E_TELEMETRY_DISABLED=1` and send nothing.
+
+## Where it goes
+
+Events go to a PostHog project we operate, in one request to
+`eu.i.posthog.com`. PostHog keeps no person profile for the ids and stores
+no request address; every event also asks it to skip its GeoIP step, so no
+country, city, or coordinates are derived from where the request came from.
+The data is read in aggregate. Telemetry falls under the
+[security policy](https://github.com/tester-army/e2e/blob/main/SECURITY.md).
+[Outbound connections](/security#outbound-connections) lists every other
+connection a run can make.
+
+## Feedback
+
+[`e2e feedback`](/reference/cli#e2e-feedback) is not telemetry. It sends
+nothing until you, or the coding agent in your shell, run it, and then it
+sends the words written into its flags as one `e2e_feedback` event to the
+same PostHog project, with the machine facts above. Because it is an explicit
+request, a saved `e2e telemetry disable` does not stop it, but
+`E2E_TELEMETRY_DISABLED=1` and `DO_NOT_TRACK=1` do: an environment that set
+either sends nothing, feedback included. `E2E_TELEMETRY_DEBUG=1` and
+`--dry-run` print it instead of sending it. When telemetry is on, the
+event carries the same anonymous id as your telemetry events. Secret-named
+environment variable values and well-known token shapes are redacted before
+it is sent.
+
+## The notice
+
+The first command prints a notice on stderr explaining how to opt out.
+`e2e init` is exempt, so a scaffold starts with its prompts rather than a
+disclaimer; the first command after it prints the notice. `e2e telemetry`
+is exempt too, since it is where you act on the notice. The notice leaves
+`--reporter json` output intact. The notice appears again after a
+material change to the data collected.

package/docs/tools.mdx

@@ -0,0 +1,150 @@
+---
+title: Project tools
+description: Add tools for test APIs, data setup, and device operations.
+---
+
+Put tools in an agent's `tools` so the agent can call your test API or perform
+other project-specific operations during a goal. Wrap each AI SDK tool with
+`defineTool` to declare whether it changes state and which platforms support it.
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { defineTool } from 'e2e/agent';
+import { web } from '@e2e-dev/web';
+import { gateway, tool } from 'ai';
+import { z } from 'zod';
+
+const APP_URL = process.env.APP_URL ?? 'http://localhost:3000';
+
+const seedCart = defineTool(
+  tool({
+    description: 'Seed the cart with a SKU via the store test API',
+    inputSchema: z.object({ sku: z.string() }),
+    execute: async ({ sku }) => {
+      const response = await fetch(`${APP_URL}/api/test/cart?sku=${encodeURIComponent(sku)}`);
+      if (!response.ok) throw new Error(`Cart seeding failed: HTTP ${response.status}`);
+      return response.text();
+    },
+  }),
+  { mutates: true },
+);
+
+export default {
+  targets: [{ engine: web(), app: { url: APP_URL } }],
+  agents: { default: { model: gateway('openai/gpt-6-luna-fast'), tools: { seedCart } } },
+} satisfies E2EConfig;
+```
+
+The agent can now seed a cart before checking out:
+
+```ts
+await agent.act('seed the cart with SKU {sku}, then check out', {
+  params: { sku: 'SHIRT-BLUE-M' },
+});
+```
+
+Built-in tool names are reserved. Loading the config fails with
+`INVALID_CONFIG` for a project tool named like a grammar verb (`tap`,
+`screenshot`), a tool the harness adds (`complete_step` in every step,
+`locate`, `start_recording`, and `stop_recording` in an `e2e mcp` session,
+`report_finding` under `e2e explore`), and for a value `defineTool` did not
+build. Tools belong to
+the built-in agent: an entry with a custom `executor` cannot also set
+`tools`.
+
+## Declare what a tool does
+
+`defineTool` takes the AI SDK tool and these annotations:
+
+| Annotation | Meaning |
+| --- | --- |
+| `mutates: true` | Changes state. The call counts against the step's action budget and runs in order with the other actions. The [replay cache](/cache) cannot replay it, so the agent runs that part of the step live. |
+| `mutates: false` | Read-only. Runs without an action slot and may observe the screen. |
+| `platforms: ['ios', 'android']` | Offers the tool only on those platforms. Omit this to offer it on every target. |
+
+A failed call still consumes its action slot. The runner checks the budget
+before dispatch, including when the model requests several calls at once.
+
+## Observe from a tool
+
+A read-only tool can observe the redacted screen. This example saves a
+masked screenshot as report evidence when pixels are available:
+
+```ts
+import { randomUUID } from 'node:crypto';
+import { defineTool, getToolContext } from 'e2e/agent';
+import { tool } from 'ai';
+import { z } from 'zod';
+
+const captureEvidence = defineTool(
+  tool({
+    description: 'Save a screenshot as evidence in the report',
+    inputSchema: z.object({}),
+    execute: async (_input, executionOptions) => {
+      const context = getToolContext(executionOptions);
+      const observation = await context.observe({ pixels: true });
+      const pixels = observation.pixels;
+      if (pixels === undefined) {
+        return { withheld: observation.pixelsWithheld ?? 'UNSUPPORTED_CAPABILITY' };
+      }
+      const id = await context.attachScreenshot(pixels, `evidence-${randomUUID()}`);
+      return { evidence: id };
+    },
+  }),
+  { mutates: false },
+);
+```
+
+`observe()` returns a redacted text snapshot. Request `tree: true` for the
+node tree or `pixels: true` for a masked screenshot. The built-in agent's
+restrictions apply, including no pixels
+[after a secret fill](/security#screenshots-after-a-secret-fill).
+
+Observe in a separate read-only tool. A mutating tool already occupies the
+action queue, so observing from inside it would wait on its own operation.
+
+`attachScreenshot(pixels, label)` adds the image to the step's artifacts and
+returns its artifact id. Use a unique label for each capture to avoid
+overwriting an earlier image.
+
+## Device toolkits
+
+`@e2e-dev/mobile` ships a ready tool pack for its engine. Pass every device
+engine to one `mobileTools` call, and spread the pack to add your own tools
+beside it:
+
+```ts
+import type { E2EConfig } from 'e2e';
+import { mobile } from '@e2e-dev/mobile';
+import { mobileTools } from '@e2e-dev/mobile/tools';
+import { gateway } from 'ai';
+
+const iphone = mobile({ platform: 'ios' });
+const pixel = mobile({ platform: 'android' });
+
+export default {
+  targets: [
+    { name: 'iphone', engine: iphone, app: { bundleId: 'Settings' } },
+    { name: 'pixel', engine: pixel, app: { bundleId: 'com.android.settings' } },
+  ],
+  agents: {
+    default: {
+      model: gateway('openai/gpt-6-luna-fast'),
+      tools: { ...mobileTools(iphone, pixel) /* , ...your own tools */ },
+    },
+  },
+} satisfies E2EConfig;
+```
+
+The pack adds `open_app`, `swipe`, and `alert` on device targets.
+Screenshots and point actions come from the engine's built-in tools.
+See the [Mobile engine reference](/reference/mobile#agent-tools).
+
+<CardGroup cols={2}>
+  <Card title="Custom executors" icon="gears" href="/executors">
+    Replace the prompt or decision loop.
+  </Card>
+  <Card title="How agent steps work" icon="stairs" href="/agent-steps">
+    What the model sees and what it may do.
+  </Card>
+</CardGroup>

package/docs/viewports.mdx

@@ -0,0 +1,146 @@
+---
+title: Testing viewports
+description: Run a suite at several sizes, resize inside a test, and know what the agent sees at each.
+---
+
+Set a viewport on the target to run a suite at that size. Use
+`browser.setViewport()` to resize during a test. The agent uses the current size
+and cannot resize it.
+
+## One size per target
+
+`viewport` sets the initial size of each attempt, in pixels. The default is
+1280 by 720. Add one target for each size you want to test:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { web } from '@e2e-dev/web';
+
+const app = { url: 'http://localhost:3000' };
+
+export default {
+  targets: [
+    { name: 'desktop', engine: web(), app },
+    { name: 'tablet', engine: web({ viewport: { width: 820, height: 1180 } }), app },
+    { name: 'phone', engine: web({ browser: 'webkit', viewport: { width: 390, height: 844 } }), app },
+  ],
+} satisfies E2EConfig;
+```
+
+Each test runs once per target, with a separate result labeled `desktop`,
+`tablet`, or `phone`. Run one size with `npx e2e run --target phone`, or list
+the selected test-target pairs with `npx e2e list`.
+
+To have the runner start one dev server for every size, give every target
+the same `app.command`: targets declaring one command share one process; see
+[several targets](/starting-your-app#several-targets). Every retry starts at
+the target's configured size.
+
+## Fill the window
+
+`viewport: null` sets no size, so the page fills the browser window. The
+agent, screenshots, and videos use that size. Use it in a headed run, or
+with a hosted browser whose live view would show a fixed 1280 by 720 page
+in one corner:
+
+```ts title="e2e.config.ts"
+{ name: 'window', engine: web({ viewport: null }), app }
+```
+
+The page takes the window's size, minus the toolbar in a headed browser. A
+local headed launch opens a small window; a hosted browser uses the window
+size you requested from the service. `browser.setViewport` still works.
+
+## Resize inside a test
+
+`browser.setViewport` resizes the page for the rest of the attempt, through
+`app.restart()` and `app.clearState()`:
+
+```ts
+import { test } from '@e2e-dev/web';
+import { expect } from 'e2e';
+
+test('the menu collapses on a narrow screen', async ({ app, screen, browser }) => {
+  await app.open('/');
+  await expect(screen.getByRole('navigation')).toBeVisible();
+
+  await browser.setViewport({ width: 390, height: 844 });
+  await expect(screen.getByRole('button', 'Menu')).toBeVisible();
+  await expect(screen.getByRole('navigation')).toBeHidden();
+});
+```
+
+Use this when one test needs to check a layout before and after resizing.
+For a whole suite, configure separate targets instead.
+
+Call it before `app.open` to load the first page at that size, so the app
+never renders its desktop layout first:
+
+```ts
+await browser.setViewport({ width: 390, height: 844 });
+await app.open('/');
+```
+
+The resize appears as a step in the report. To check the exact width, read
+it from the page:
+
+```ts
+const width = await browser.evaluate(() => window.innerWidth);
+expect(width).toBe(390);
+```
+
+## What the agent sees
+
+The agent receives the current viewport size with each observation.
+Screenshots cover that viewport. Off-screen elements can still appear in
+the tree, and tapping them scrolls them into view. The agent can scroll to
+reveal lazy-loaded content.
+
+Resize before calling `agent.act`:
+
+```ts
+test('the mobile menu reaches pricing', async ({ app, agent, browser }) => {
+  await browser.setViewport({ width: 390, height: 844 });
+  await app.open('/');
+  await agent.act('open the navigation menu and go to Pricing');
+  await expect(browser).toHaveURL('/pricing');
+});
+```
+
+## Scoping a test to one size
+
+`platforms` and `requires` cannot select a viewport; all Playwright targets
+use platform `web`. To skip a test at some sizes, check the width in the
+test:
+
+```ts
+test('the sidebar stays open', async ({ app, screen, browser }) => {
+  await app.open('/');
+  test.skip((await browser.evaluate(() => window.innerWidth)) < 768, 'desktop layout only');
+  await expect(screen.getByRole('complementary')).toBeVisible();
+});
+```
+
+To keep a whole group of size-specific tests apart, give them their own
+config and run it with `--config`.
+
+## Limits
+
+A narrow viewport tests responsive layout. It still uses a desktop browser
+with a scale factor of 1, desktop pointer behavior, and its usual user agent.
+It does not emulate touch, color scheme, or reduced-motion preferences.
+Use the [device engine](/mobile) for tests on a simulator or emulator.
+
+A video keeps the size the page had when it started recording; set
+`web({ screencast: { size } })` for one fixed size. A browser provider that
+records, such as [Kernel](/integrations/kernel#recordings), records the whole
+screen instead.
+
+<CardGroup cols={2}>
+  <Card title="Working with the browser" icon="globe" href="/browser">
+    Routes, cookies, dialogs, downloads, and the rest of the `browser` fixture.
+  </Card>
+  <Card title="Web engine reference" icon="masks-theater" href="/reference/web">
+    Every engine option and `browser` method, with timeouts and errors.
+  </Card>
+</CardGroup>