e2e 0.15.1 → 0.16.0

package/docs/security.mdx

@@ -0,0 +1,308 @@
+---
+title: Security model
+description: What the model can access, how the runner handles secrets, and the limits of its protections.
+---
+
+Test code and configuration run with your operating-system permissions.
+This includes engines, tools, reporters, stores, secret providers, and app
+commands. e2e does not sandbox them. Run untrusted pull request code in an
+external sandbox without secrets or write tokens.
+
+The model receives app content as untrusted evidence. It requests actions
+through tools; the runner validates and performs those actions. Cached
+actions also run with your permission, so review shared cache entries as
+you would test code.
+
+## What the model sees
+
+Model requests separate runner policy and trusted project context from app
+content. App content cannot grant tools, credentials, or a larger budget.
+[How agent steps work](/agent-steps) explains what a step shows the model and
+what it may do; this page lists the trust boundary.
+
+| In the prompt | Form |
+| --- | --- |
+| Your instruction and params | Quoted inside `<instruction>` and marked as data. A `Secret` param is projected to its name and purpose. |
+| The screen | An accessibility tree with node ids, roles, names, text, test ids, link targets, and states, cut at a size limit. No selectors. A link's query and fragment, where tokens ride, show only as `?…` and `#…`. |
+| The location | The current URL, redacted. |
+| A screenshot | When needed or requested, only if the engine proves secure fields are masked and no secret has been filled in the attempt. |
+| Prior steps | `agent.act` receives bounded records from the same test or serial group. Judgments use the current screen and instruction, without prior-step records. |
+
+Secure fields appear as `value=<secure>`. Known secret values in model input
+and reports become `<secret:name>`. This [redaction](#redaction) has limits
+for transformed values, images, and app logs.
+
+## What the model may do
+
+Built-in tools have closed schemas and are offered only when the engine
+supports them. Project tools must use `defineTool` and cannot replace a
+built-in name. Only read-only project tools may request an observation.
+
+A call to a tool the step does not offer, or with an argument its schema does
+not declare, never runs: the model reads the refusal and tries again. A node
+id that is not on the current screen fails the action with
+`LOCATOR_NOT_FOUND`. The runner never evaluates model text as code,
+selectors, shell commands, or config.
+
+A tool call that breaks a rule fails with `POLICY_DENIED`:
+
+| Denied | Why |
+| --- | --- |
+| Navigation to `file:`, `data:`, or `javascript:` | These URL schemes are forbidden. |
+| A malformed URL | Nothing is repaired. |
+| A `Secret` reaching a JSON sink | Handles carry no value, and no value may be serialized. |
+| Reading a value, text, or attribute from a secure field, with a getter or a `toHaveValue`, `toHaveText`, `toContainText`, or `toHaveAttribute` matcher | Assert the outcome, not the secret. A withheld value is redacted, not empty or absent. |
+| `type_secret` with an undeclared name | Only secrets in the step's params can be filled. |
+| `type_secret` into a disabled or non-editable node | Sinks are `textbox`, `searchbox`, and `combobox`. |
+| A password into a field that is not a password field | Purpose must match; a generic secret fills any editable input. |
+| A project tool not made with `defineTool`, or without `execute` | Undeclared semantics are not trusted. |
+| A mutating project tool requesting an observation | Observe in a separate call. |
+| `vision: 'only'` when pixels are withheld | There is nothing to fall back to. |
+| `app.screenshot()` after a secret fill | The app may display the secret outside a secure field. |
+
+## Navigation and origins
+
+Navigation and secret fills have no origin allowlist. The agent can follow
+the app to other sites and use the secrets supplied to the current step
+there. Password fills still require a password field.
+
+The browser engine uses a hostname-based site check for two purposes:
+
+- Configured `headers` are added to requests on the target's site.
+- Child frames outside that site, and `data:` frames, are omitted from
+  observations.
+
+The site check approximates the registrable domain without a public suffix
+list. Shared hosting domains such as `vercel.app` count as one site, so
+headers can reach other hosts under that domain. `basicAuth` is broader:
+it answers a `401` challenge from any origin.
+
+The runner checks forbidden schemes on explicit navigation: `app.open`, the
+agent's `navigate` verb, and `device.openLink`. `device.openApp` and the
+`open_app` tool take an app id and refuse a link, since the device would
+open one as a URL. The runner does not re-check redirects or attach to popup
+pages. Register `browser.onDialog` to handle browser dialogs.
+
+## Secrets
+
+A `Secret` is an opaque handle with no plaintext accessor. Three sinks
+accept one: a locator's `fill`, the params of an agent step, and an engine
+option that declares it (`web({ basicAuth: { password } })`). The
+[Signing in](/authentication) guide covers usage; the rules are here.
+
+Before a model-directed fill, the runner checks that the step declared the
+secret, the secret is configured, and the target is an enabled editable
+input. Passwords require a password field. It then resolves the value on the
+host and passes it to the engine. Replay repeats these checks.
+
+A deterministic `fill(secret)` uses the field chosen by test code without
+those model-directed field checks. It still registers the value for
+redaction and disables pixels for the rest of the attempt.
+
+### Screenshots after a secret fill
+
+After any secret fill, for the rest of the attempt, screenshot and pixel
+tools are unavailable, `app.screenshot()` is denied, and the runner omits
+assertion and failure screenshots. The app could display the value anywhere,
+beyond the field's masked rectangle.
+
+A test that restores a [session](#sessions) whose setup filled a secret keeps
+this protection. A setup that signs in without filling a secret, for example
+by setting a session cookie with `browser.setCookies`, leaves screenshots
+available to the tests that restore it. A value it uses that is not a
+configured secret is not redacted.
+
+### Secrets in engine options
+
+A secret an engine option holds (`web({ basicAuth: { password: secrets.get(name) } })`)
+is resolved on the host when each attempt starts and registered for redaction
+then, together with what the engine derives from it (for basic auth, the
+base64 `user:password` the `Authorization` header carries). It is not a fill
+and is protected as text only: reports, `screen.txt`, failure pages, what the
+model and an `e2e mcp` session read, text downloads, and the Playwright
+trace's text (the options the browser opened with, and the `Authorization:
+Basic` header of every request) are redacted, while screenshots, the trace's
+screencast frames, and the model's pixels are kept. A page that renders the
+password on screen is not masked in pixels.
+
+### Redaction
+
+The runner replaces every occurrence of a registered secret value with
+`<secret:name>` in model input, reports, a worker process's console output,
+and the trace text of a session a secret was filled on or an engine option
+holds. That includes a value the test passes as a plain string rather than
+a `secrets.get()` handle: test and `describe` titles, step labels (an
+`app.open` URL, a locator's text), an `agent.act` or `agent.assert`
+instruction, the strings in `agent.act` params, `agentContext`, and an
+`e2e explore` goal. Titles are redacted as the test file is collected, so
+the test id, `--grep`, the reporters, and the artifact and failure-page file
+names all see `<secret:name>`, and two titles in one file that differ only
+in a secret value are a duplicate title path. A custom executor's
+`ctx.step` and the built-in agent's prompt carry the marker, so only a
+handle can type the real value.
+
+Matching covers the value as written, JSON-escaped, HTML-escaped, and
+percent-encoded, in any letter case (so a CSS `text-transform` does not hide
+it), and with its whitespace collapsed. Text cut at a length limit is masked
+too when it ends with the first 8 or more characters of a value (half of a
+value shorter than 16). Redaction rewrites only what the runner reports;
+assertions still compare the text the page shows. Secret values never enter
+cache entries.
+
+A static value shorter than 6 characters is `INVALID_CONFIG`, and a provider
+returning one fails the fill: every occurrence of so short a value would take
+ordinary text with it.
+
+### Where redaction stops
+
+- A transformed form of the value, such as its last four characters, is not
+  the secret and passes through.
+- A value encoded as a whole rather than character by character, base64 or
+  a hash, is a different string and passes through.
+- A value the app renders in another Unicode normalization form, decomposed
+  accents where the secret has composed ones, passes through.
+- Text that widens the whitespace inside the value, or drops it, passes
+  through.
+- A value spread over several nodes, one character per cell as a PIN pad
+  shows it, is not one string in the tree and passes through.
+- Text cut at a length limit is matched by its prefix only in the value's
+  own form: a case mapping that changes length (`ß` shown as `SS`) or
+  whitespace collapsed to one space escapes the prefix rule.
+- A title is redacted when its file is collected, before any provider
+  runs, so a value a secret provider returns is not redacted from titles.
+- A Playwright trace from a session no secret reached is kept as recorded,
+  so a value the test only spelled into a URL or a locator stays in its
+  text.
+- When tests run in the runner's own process, with the config or tests
+  passed in memory by an embedding host, their console output is not
+  redacted.
+- A Playwright trace from a session a secret reached, filled or held by the
+  engine, is rewritten before it is kept: any run of 8 or more characters of
+  a value becomes `<secret:name>`, so unrelated text that shares 8 characters
+  with a value is masked too. A base64 or base64url run that decodes to a
+  value or such a fragment (a basic-auth header, a cookie, a token segment)
+  is replaced whole. After a fill, screencast frames are dropped; with only
+  an engine-held secret they are kept. An image the app served that
+  draws the secret survives. A trace the runner cannot rewrite is deleted and
+  the attempt records `TRACE_WITHHELD`.
+- A video masks nothing and is labeled `redaction: "incomplete"`. It is
+  never recorded unless you ask. A recording a hosted browser or
+  device service keeps stays with that service; the report holds only its
+  URL.
+- A download is what the app served. Once a secret reached the session,
+  filled or held by the engine, a text download (`text/*` or JSON by file
+  extension) is rewritten the way trace text is and labeled
+  `redaction: "complete"`. Any other download, a binary or one from a
+  session no secret reached, is kept as served and labeled
+  `redaction: "incomplete"`.
+- The `headers` a protected preview needs are recorded in the trace as
+  request headers. Share that trace as you would the bypass secret.
+- `command.log` is captured as the process writes it.
+  The runner does not redact them. Keep secrets out of your app's stdout.
+
+## Sessions
+
+Saved sessions contain authentication state. They are encrypted and last
+for one run. A session also carries the secret values the setup test resolved
+from a provider before saving it, inside the same ciphertext, and whether a
+secret was filled on it. The test that restores the session redacts those
+values and keeps its pixels withheld, so a stored token the app echoes back
+never reaches a failure message, a report, or the model. The envelope names
+the secrets in the clear, authenticated; the values are never written
+unencrypted.
+
+| Property | Value |
+| --- | --- |
+| Location | `sessions/<runId>/` under the [output directory](/reference/config#output) (default `.e2e/sessions/<runId>/`), directory mode `0700`, files `0600` |
+| Cipher | AES-256-GCM, a fresh 12-byte IV per save |
+| Key | 32 random bytes generated per run and held in memory. Workers receive it over IPC, never through disk or the environment. |
+| Bound to | Run id, target, engine name and version, platform, and app identity, all authenticated as additional data |
+| Lifetime | At most 24 hours, sooner if the saved state declares an earlier expiry |
+| Cleanup | The directory is removed when the run ends |
+
+Invalid, expired, missing, or mismatched sessions are refused. See the
+[session errors](/reference/errors) for the corresponding codes.
+
+## Cache trust
+
+Shared cache entries deserve the same review as test code. They can replay
+actions without consulting a model and are not signed.
+
+The runner validates entries before use. Invalid entries become misses.
+Files are named by a hash of the expected key and refused if larger than
+1 MiB. Entries store actions and a redacted summary; replay executes only
+the actions. Secret fills store a name and authorize the fill again.
+
+`e2e init` ignores `.e2e/cache/` in Git. If you choose to
+[commit recordings](/cache#commit-the-replay-cache), review their actions and typed
+values. CI defaults to read-only when no cache mode is configured.
+
+## Artifacts and logs
+
+By default everything a run writes lands under `.e2e/`. The
+[`output`](/reference/config#output) option or `--output` moves it, the
+replay cache stays at `cache.dir`, and `command.log` is whatever
+project-relative path you configure.
+
+| Path | Contents | Redaction |
+| --- | --- | --- |
+| `report.json` | The run record. Errors, reasoning excerpts, and action summaries pass the redactor. | complete |
+| `artifacts/.../*.png` | Screenshots. Secure fields are masked before screenshots are returned or saved as artifacts. | complete |
+| `artifacts/.../trace.zip` | Playwright trace, rewritten after a secret fill or when an engine option holds a secret; screencast frames are dropped after a fill only. | not-required, complete, or deleted |
+| `artifacts/.../*.webm`, `*.mp4` | Video, opt-in through `video` or `--video`; `retain-on-failure` keeps only the attempts that did not pass. | incomplete |
+| `artifacts/.../downloads/*` | Files the app served through a download. Text ones are rewritten after a secret fill or when an engine option holds a secret; the rest are kept as served. | complete or incomplete |
+| `artifacts/.../failure/screen.txt` | The redacted tree at the moment of failure. | complete |
+| `ai-trace.json` | Every model call, opt-in via `--ai-trace`. Prompts are the redacted ones the model saw; inline image bytes are replaced by their size. | as the prompt |
+| `sessions/<runId>/` | Encrypted sessions, deleted at cleanup. | encrypted |
+| `videos/<session>/*.webm`, `*.mp4` | Videos an `e2e mcp` session recorded with `start_recording`, kept until you delete them. Never handed to `artifacts.store`. | none |
+| `cache.dir` (default `.e2e/cache/`) | Replay cache entries. Not under `output`. | secret-free by construction |
+| The path in `command.log` | Your app's stdout and stderr, verbatim. Must be inside the project root. | none |
+
+An `artifacts.store` receives each artifact's bytes as they are produced,
+after trace and download redaction, with the `redaction` label the report
+records, so a store can export only what the runner vouches for. A
+`cache.store` receives every recorded entry and supplies every replayed one;
+what comes back is validated again before use.
+
+## Outbound connections
+
+Built-in features may connect to:
+
+- one telemetry request per CLI invocation to `eu.i.posthog.com`, plus at
+  most one per session that `e2e mcp` serves, unless opted out; see [Telemetry](/telemetry)
+- one request to `eu.i.posthog.com` per `e2e feedback` you run; see
+  [Feedback](/telemetry#feedback)
+- the provider endpoint used by your configured model. Deterministic tests
+  and fully replayed action steps make no model call.
+- readiness probes against the `readyUrl` of the app the runner starts
+- a Playwright browser download, once, when the browser is missing
+- the DevTools endpoint you name in `web({ connect })`
+- the GitHub API, only from the opt-in `@e2e-dev/github` reporter
+- the Kernel API and its hosted browsers, only from
+  [`kernel()`](/integrations/kernel)
+- the Expo API and the simulators' agent-device daemons, only from
+  [`easSimulators()`](/integrations/eas). It authenticates with `EXPO_TOKEN`,
+  else with the eas-cli login in `~/.expo/state.json`: on a CI runner, set
+  `EXPO_TOKEN` or keep that login out of the runner's home
+- the app under test, and whatever that app itself loads
+
+There is no crash reporting or update check. `--ai-trace` writes a local
+file without uploading it. Your test code, plugins, and app can make
+additional connections.
+
+## MCP and explore
+
+`e2e mcp` uses stdio and the same action authorization as a test run. It
+allows four open sessions at once by default
+([`--max-sessions`](/reference/mcp), 1 through 16) and closes idle sessions.
+
+`e2e explore` offers every configured credential to every step; see
+[Exploring without a test](/explore#your-config-applies).
+
+## Reporting a vulnerability
+
+Report privately through
+[GitHub security advisories](https://github.com/tester-army/e2e/security/advisories/new)
+or by mail to security@tester.army. The
+[security policy](https://github.com/tester-army/e2e/blob/main/SECURITY.md)
+states response times and supported versions.

package/docs/starting-your-app.mdx

@@ -0,0 +1,231 @@
+---
+title: Starting your app
+description: Point the runner at a running app, or let it start the app for the run.
+---
+
+Set your app's URL in the target's `app`. If you also provide a start
+command, the runner starts the app, waits until it is ready, and stops it
+when the run ends.
+
+## Point at a running app
+
+Import `web` from `@e2e-dev/web` and put this in the exported
+config object:
+
+```ts title="e2e.config.ts"
+export default {
+  targets: [{
+    engine: web(),
+    app: { url: process.env.APP_URL ?? 'http://localhost:3000' },
+  }],
+} satisfies E2EConfig;
+```
+
+`app.open('/path')` resolves against `app.url`. This config reads `APP_URL`, so
+you can change the address for a run without editing the file. URLs without
+a scheme use HTTPS, or HTTP for loopback hosts.
+
+## Let the runner start it
+
+Add `app.command` and the runner spawns the dev server, waits until `app.url`
+answers, and stops it when the run ends, fails, or is interrupted:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { web } from '@e2e-dev/web';
+import { gateway } from 'ai';
+
+export default {
+  agents: {
+    default: {
+      model: gateway('openai/gpt-6-luna-fast'),
+      system: 'You are a thorough QA agent. Verify every outcome.',
+    },
+  },
+  targets: [{
+    engine: web(),
+    app: {
+      url: process.env.APP_URL ?? 'http://localhost:3000',
+      command: {
+        executable: 'npm',
+        args: ['run', 'dev'],
+        reuseExisting: true,
+        log: '.e2e/logs/app.log',
+      },
+    },
+  }],
+} satisfies E2EConfig;
+```
+
+```text
+ ✓ target "web" command ready 1.20s
+```
+
+The main command options are:
+
+| Option | Purpose |
+| --- | --- |
+| `reuseExisting: true` | Use an already-running server locally. CI ignores this and reports `APP_ALREADY_RUNNING`. |
+| `log` | Save the app's stdout and stderr to a project-relative file. Startup errors include the last 20 lines, with `command.env` values redacted. The log file itself is unredacted. |
+| `env` | Environment variables for the app. The child inherits `PATH`, `HOME`, and the temp-directory variables, plus these. |
+| `startupTimeout` | How long to wait for readiness, 60 seconds by default. |
+
+To probe a health endpoint instead of the base URL, set `readyUrl` beside
+`command`: `app: { url, command, readyUrl: 'http://localhost:3000/health' }`.
+
+The command runs without a shell. `executable` is resolved on `PATH` and
+`args` are passed as written. The child inherits only `PATH`, `HOME`, the
+temp-directory variables, and `env`, so pass what the app needs explicitly.
+See the
+[command reference](/reference/config#command) for all options.
+
+### Let the runner pick the port
+
+Use port `0` to run multiple checkouts without a port conflict. The runner
+chooses a free port and replaces `{port}` in the command's `args` and `env`.
+Your app command must use that value:
+
+```ts
+engine: web(),
+app: {
+  url: 'http://127.0.0.1:0',
+  command: { executable: 'pnpm', args: ['dev', '--port', '{port}'], env: { PORT: '{port}' } },
+},
+```
+
+Tests can read the allocated URL from `app.baseUrl`. Cache entries remain
+valid when the port changes. Use `127.0.0.1` or `[::1]`, matching the address
+the command binds; `localhost:0` is rejected. See
+[free ports](/reference/config#free-ports) for details.
+
+### Next.js dev targets
+
+Next.js 16 blocks cross-origin requests to its dev resources. A target that
+opens `127.0.0.1` or `[::1]` while `next dev` identifies as `localhost`
+renders and never hydrates. Locators resolve and clicks land, but no React
+handler runs, so the test fails and every retry fails the same way.
+
+Add the host the target opens to `allowedDevOrigins`:
+
+```ts title="next.config.ts"
+import type { NextConfig } from 'next';
+
+const nextConfig: NextConfig = {
+  allowedDevOrigins: ['127.0.0.1'],
+};
+
+export default nextConfig;
+```
+
+Use `'[::1]'` in the list when `app.url` opens `http://[::1]:0`.
+
+A target on `localhost` needs no entry, because `next dev` identifies as
+`localhost`. A production `next start` target is unaffected either way.
+
+## Dependencies
+
+<Note>
+This version starts only `app.command` (one process for every target that
+declares the same command). Databases, mocks, and other processes the app
+needs are not started by the runner yet: bring them up before the run, or
+have `app.command` start a script that brings them up and serves the app. A
+services API returns in a later release.
+</Note>
+
+## Several targets
+
+Two browsers on one app are two targets that each name it:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { web } from '@e2e-dev/web';
+
+const app = { url: 'http://localhost:3000' };
+
+export default {
+  targets: [
+    { name: 'chromium', engine: web(), app },
+    { name: 'webkit-mobile', engine: web({ browser: 'webkit', viewport: { width: 390, height: 844 } }), app },
+  ],
+} satisfies E2EConfig;
+```
+
+Two targets that declare the same `app.command` share one process. Every
+test runs on every target unless it is scoped with `platforms` or
+`requires`; `--target <name>` selects at run time. A device target sits
+beside a browser target in the same list; see the
+[Mobile engine reference](/reference/mobile). One target per screen size is
+how a suite runs at several [viewports](/viewports).
+
+## Protected previews
+
+Use `headers` to access protected previews or skip a tunnel's interstitial
+page. For example, a Vercel preview can use a protection bypass header:
+
+```ts title="Target fragment"
+engine: web({
+  headers: {
+    'x-vercel-protection-bypass': process.env.VERCEL_AUTOMATION_BYPASS_SECRET ?? '',
+  },
+}),
+app: {
+  url: process.env.PREVIEW_URL ?? 'http://localhost:3000',
+  identity: 'storefront',
+},
+```
+
+`headers` apply to requests on the target's site, determined by its hostname.
+The check also includes subdomains and treats shared suffixes such as
+`vercel.app` as one site. See [header scope](/security#navigation-and-origins)
+before using a bypass secret. For ngrok, use
+`'ngrok-skip-browser-warning': '1'`.
+
+`app.identity` gives the app a stable cache identity when its preview URL changes.
+
+Use `basicAuth` for HTTP basic authentication. The browser sends these
+credentials when any origin requests them with a `401` challenge:
+
+```ts title="Target fragment"
+engine: web({
+  basicAuth: { username: 'preview', password: process.env.PREVIEW_PASSWORD ?? '' },
+}),
+app: { url: 'https://staging.example.com' },
+```
+
+To keep the password out of reports, logs, and what the agent reads, declare
+it under `secrets` and pass `secrets.get(name)`. The engine resolves it when
+each attempt starts, and the runner redacts it like any configured secret; a
+name that `secrets` does not declare fails the config load with `INVALID_CONFIG`:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { secrets } from 'e2e';
+import { web } from '@e2e-dev/web';
+
+export default {
+  targets: [
+    {
+      engine: web({
+        basicAuth: { username: 'preview', password: secrets.get('previewPassword') },
+      }),
+      app: { url: 'https://staging.example.com' },
+    },
+  ],
+  secrets: { previewPassword: () => process.env.PREVIEW_PASSWORD ?? '' },
+} satisfies E2EConfig;
+```
+
+<Note>
+Setting `headers` disables the browser's HTTP cache and blocks service
+workers. Playwright traces include request headers, so a protected-preview
+trace can contain your bypass secret. Restrict access to those traces.
+</Note>
+
+<CardGroup cols={2}>
+  <Card title="Config reference" icon="sliders" href="/reference/config#the-app-under-test">
+    `app`, `command`, and every app key.
+  </Card>
+  <Card title="Web engine reference" icon="masks-theater" href="/reference/web">
+    Browser, viewport, `connect`, `headers`, `basicAuth`.
+  </Card>
+</CardGroup>