e2e 0.21.1 → 0.21.2

package/docs/reference/security.mdx

@@ -9,7 +9,28 @@ icon: "lock"
 The [Security](/security) guide explains the trust boundary. This page
 lists the rules behind it, as the code enforces them.
 
-## Model input
+## Reference
+
+| Section | Covers |
+| --- | --- |
+| [Model input](#model-input) | What the model reads in each step |
+| [Tool calls](#tool-calls) | Built-in and project tools, node ids |
+| [Policy denials](#policy-denials) | Calls that fail with `POLICY_DENIED` |
+| [Navigation](#navigation) | URL schemes per entry point |
+| [Site check](#site-check) | Headers, child frames, `basicAuth` |
+| [Secret fills](#secret-fills) | `type_secret`, `fill(secret)` |
+| [Screenshots after a secret fill](#screenshots-after-a-secret-fill) | Screenshots, pixel tools, restored sessions |
+| [Secrets in engine options](#secrets-in-engine-options) | `basicAuth` and other engine options |
+| [Redaction](#redaction) | Model input, reports, downloads, output |
+| [Where redaction stops](#where-redaction-stops) | Transformed values, video, `command.log` |
+| [Downloads](#downloads) | Files the app served |
+| [Sessions](#sessions) | Saved authentication state |
+| [Cache trust](#cache-trust) | Replay cache files and `cache.store` |
+| [Artifacts](#artifacts) | Files under the output directory |
+| [Outbound connections](#outbound-connections) | Destinations the runner connects to |
+| [MCP and explore](#mcp-and-explore) | MCP sessions, the explorer's sign-ins |
+
+### Model input
 
 The model reads app content as quoted evidence, kept apart from runner policy
 and trusted project context. App content can't grant tools, credentials, or a
@@ -23,22 +44,26 @@ larger budget. [How agent steps work](/agent-steps) explains a step in full.
 | A screenshot | When needed or requested, and only if the engine proves secure fields are masked and no secret was filled in the attempt. |
 | Prior steps | `agent.act` receives bounded records from the same test or serial group. Judgments use only the current screen and instruction. |
 
-## Tool calls
+### Tool calls
 
 - Built-in tools have closed schemas and appear only when the engine
   supports them.
 - Project tools must use `defineTool` with an `execute` and can't replace a
-  built-in name. Any other tool fails the config load with `INVALID_CONFIG`.
+  built-in name.
 - Only read-only project tools may request an observation.
 - A call to a tool the step doesn't offer, or with an undeclared argument,
   never runs. The model reads the refusal as the call's result and tries
   again.
-- A node id that isn't on the current screen fails the action with
-  `LOCATOR_NOT_FOUND`.
 - The runner never evaluates model text as code, selectors, shell commands,
   or config.
 
-## Policy denials
+| Fails with | When |
+| --- | --- |
+| `CONFIG_LOAD_FAILED` | A `defineTool` without an `execute`, at config load. |
+| `INVALID_CONFIG` | A tool not created with `defineTool`, or one named after a built-in or harness tool. |
+| `LOCATOR_NOT_FOUND` | A node id that isn't on the current screen. The action fails, and the model reads the failure and acts again. |
+
+### Policy denials
 
 These calls fail with `POLICY_DENIED`.
 
@@ -53,24 +78,29 @@ These calls fail with `POLICY_DENIED`.
 | A password into a field that isn't a password field | Purpose must match. A generic secret fills any editable input. |
 | A mutating project tool requesting an observation | Observe in a separate call. |
 | `vision: 'only'` when pixels are withheld | There is nothing to fall back to. |
-| `app.screenshot()` or `toHaveScreenshot()` after a secret fill | The app may show the secret outside a secure field. |
+| `browser.setCookies` with a cookie URL that isn't `http:` or `https:` | Only a web origin holds cookies. |
+| `app.screenshot()` or `toHaveScreenshot()` after a secret fill | See [Screenshots after a secret fill](#screenshots-after-a-secret-fill). |
 | `toHaveScreenshot()` when the engine can't prove every secure field is masked | The screenshot isn't kept. |
 | An agent, executor, or MCP `upload` of a file outside the project root, a hidden file, or a file under a hidden directory | Only project files can leave the computer through a form. |
 | An `act` or `vision: true` step whose semantic capture times out after a secret fill | It can't fall back to a screenshot. |
 
-## Navigation
+### Navigation
+
+| Entry point | Accepts | Refuses |
+| --- | --- | --- |
+| `app.open`, `browser.goto`, the agent's `navigate` verb | `http:`, `https:`, and the exact `about:blank` | Every other scheme |
+| `device.openLink` | Any absolute URL: an app's custom scheme, or `http:` and `https:` | `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, `filesystem:` |
+| `device.openApp`, the `open_app` tool | An app id | A link |
 
-- There is no origin allowlist. The agent can follow the app to other sites
-  and use the step's secrets there.
-- `app.open`, `browser.goto`, and the agent's `navigate` verb accept only
-  `http:`, `https:`, and the exact `about:blank`.
-- `device.openLink` accepts an app's custom scheme and refuses `file:`,
-  `data:`, `javascript:`, `view-source:`, `blob:`, and `filesystem:`.
-- `device.openApp` and the `open_app` tool take an app id and refuse a link.
 - The runner doesn't re-check redirects or attach to popup pages. Register
   `browser.onDialog` to handle browser dialogs.
 
-## Site check
+<Warning>
+There is no origin allowlist. The agent can follow the app to other sites
+and use the step's secrets there.
+</Warning>
+
+### Site check
 
 The browser engine compares hostnames to decide what counts as the target's
 site.
@@ -83,12 +113,23 @@ site.
 - `basicAuth` ignores the site check. It answers a `401` challenge from any
   origin.
 
-## Secret fills
+### Secret fills
 
 A `Secret` is an opaque handle with no plaintext accessor. Three sinks accept
 one: a locator's `fill`, the params of an agent step, and an engine option
 that declares it.
 
+```ts title="tests/stripe.e2e.ts"
+import { secrets, test } from 'e2e';
+
+test('connect Stripe', async ({ screen, agent }) => {
+  await screen.getByLabel('Secret key').fill(secrets.get('stripe-key'));
+  await agent.act('Connect Stripe with the given key', {
+    params: { apiKey: secrets.get('stripe-key') },
+  });
+});
+```
+
 | Fill | Checks before the value resolves |
 | --- | --- |
 | Model-directed (`type_secret`) | The step declared the secret. The secret is configured. The target is an enabled, editable input on the newest observation. A password goes into a password field. Replay repeats these checks. |
@@ -97,7 +138,7 @@ that declares it.
 The runner resolves the value on the host and passes it to the engine. The
 model never sees or picks it.
 
-## Screenshots after a secret fill
+### Screenshots after a secret fill
 
 After any secret fill, for the rest of the attempt:
 
@@ -116,11 +157,29 @@ setting a session cookie with `browser.setCookies`, leaves screenshots on for
 the tests that restore it. A value it uses is redacted only if it's a
 configured secret.
 
-## Secrets in engine options
+### Secrets in engine options
 
 A secret in an engine option, such as
 `web({ basicAuth: { password: secrets.get(name) } })`, isn't a fill.
 
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { secrets } from 'e2e';
+import { web } from '@e2e-dev/web';
+
+export default {
+  targets: [
+    {
+      engine: web({
+        basicAuth: { username: 'preview', password: secrets.get('previewPassword') },
+      }),
+      app: { url: 'https://staging.example.com' },
+    },
+  ],
+  secrets: { previewPassword: () => process.env.PREVIEW_PASSWORD ?? '' },
+} satisfies E2EConfig;
+```
+
 - The runner resolves it on the host when each attempt starts.
 - It registers the value for redaction, together with what the engine
   derives from it. For basic auth, that is the base64 `user:password` in the
@@ -130,7 +189,7 @@ A secret in an engine option, such as
 - Screenshots and the model's pixels stay on. A page that shows the password
   on screen isn't masked in pixels.
 
-## Redaction
+### Redaction
 
 The runner replaces every registered secret value with `<secret:name>` in:
 
@@ -159,12 +218,13 @@ on.
 | Assertions | Unaffected. Redaction rewrites only what the runner reports. |
 | Cache entries | Secret values and their cut parts never enter them. |
 
-## Where redaction stops
+### Where redaction stops
 
 Redaction matches strings. These pass through:
 
 - a transformed value, such as its last four characters,
-- a value encoded as a whole, such as base64 or a hash,
+- a value encoded as a whole, such as a hash, or base64 outside a text
+  [download](#downloads),
 - a value in another Unicode normalization form, such as decomposed accents
   where the secret has composed ones,
 - text that drops the whitespace inside the value. Text that widens it passes
@@ -183,11 +243,11 @@ Redaction matches strings. These pass through:
   service, and the report holds only its URL,
 - `command.log`: the runner captures your app's output as written.
 
-## Downloads
+### Downloads
 
 A download is what the app served. When the session knows a secret value,
-the runner rewrites a text download (`text/*` or JSON, by file extension) and
-labels it `redaction: "complete"`. It replaces with `<secret:name>`:
+the runner rewrites a text download (`.txt`, `.csv`, `.json`, `.html`, or
+`.htm`) and labels it `redaction: "complete"`. It replaces with `<secret:name>`:
 
 - every whole occurrence of a value,
 - any run of 8 or more of its characters, whitespace collapsed or not, so
@@ -195,16 +255,19 @@ labels it `redaction: "complete"`. It replaces with `<secret:name>`:
 - a base64 or base64url run that decodes to either.
 
 Every other download is kept as served and labeled
-`redaction: "incomplete"`: a binary, one that isn't valid UTF-8 or can't be
-rewritten, and any download from a session that knows no secret value.
+`redaction: "incomplete"`:
+
+- any other extension,
+- one that isn't valid UTF-8 or can't be rewritten,
+- any download from a session that knows no secret value.
 
-## Sessions
+### Sessions
 
 A saved session holds authentication state. It also carries, in the same
-ciphertext, the secret values the setup test resolved from a provider and
-whether a secret was filled. The test that restores it redacts those values
-and keeps its pixels withheld. The envelope names the secrets in the clear,
-authenticated.
+ciphertext, the secret values the setup test resolved from a provider. The
+test that restores it redacts those values and keeps its pixels withheld. The
+envelope holds, unencrypted but authenticated, the secret names and whether a
+secret was filled.
 
 | Property | Value |
 | --- | --- |
@@ -218,7 +281,7 @@ authenticated.
 The runner refuses an invalid, expired, missing, or mismatched session. See
 the [session errors](/reference/errors).
 
-## Cache trust
+### Cache trust
 
 Cache entries replay actions without a model and aren't signed.
 
@@ -235,7 +298,7 @@ Cache entries replay actions without a model and aren't signed.
 | Git | `e2e init` ignores `.e2e/cache/`. If you [commit recordings](/cache#commit-the-replay-cache), review their actions and typed values. |
 | `cache.store` | Receives every recorded entry and supplies every replayed one. What comes back is validated again. |
 
-## Artifacts
+### Artifacts
 
 A run writes under `.e2e/` by default. The
 [`output`](/reference/config#output) option or `--output` moves it. The
@@ -259,11 +322,11 @@ project-relative path you configure.
 An `artifacts.store` receives each artifact's bytes as they're produced,
 after download redaction, with the `redaction` label the report records.
 
-## Outbound connections
+### Outbound connections
 
 | Destination | When |
 | --- | --- |
-| `eu.i.posthog.com` | One telemetry request per CLI invocation, plus at most one per session `e2e mcp` serves, unless you opt out ([Telemetry](/telemetry)). One request per `e2e feedback` you run. |
+| `eu.i.posthog.com` | One telemetry request per CLI invocation or per `run()` from `e2e/runner` that returns a report, plus at most one per session `e2e mcp` serves, unless you opt out ([Telemetry](/telemetry)). One request per `e2e feedback` report sent: none under `--dry-run` or `E2E_TELEMETRY_DEBUG`, for a bug without `--repro`, or under `E2E_TELEMETRY_DISABLED` or `DO_NOT_TRACK`. |
 | Your model provider | Agent steps that call a model. Deterministic tests and fully replayed steps make no model call. |
 | A subscription provider's sign-in and model-list endpoints | `e2e login` and `e2e models` ([Subscriptions](/subscriptions)). |
 | The app's `readyUrl` | Readiness probes for an app the runner starts. |
@@ -271,17 +334,20 @@ after download redaction, with the `redaction` label the report records.
 | The DevTools endpoint in `web({ connect })` | Every run of that target. |
 | The GitHub API | Only from the `@e2e-dev/github` reporter. |
 | `registry.npmjs.org` | One request per `e2e feedback` you run, to check for a newer e2e release. Skipped with `--allow-outdated` and under `E2E_TELEMETRY_DISABLED` or `DO_NOT_TRACK`. |
+| `api.limrun.com` (or `LIMRUN_BASE_URL`) and its devices | Only from [`limrun()`](/integrations/limrun). It signs in with `LIMRUN_API_KEY`. |
 | The Kernel API and its hosted browsers | Only from [`kernel()`](/integrations/kernel). |
 | The Expo API and the simulators' agent-device daemons | Only from [`easSimulators()`](/integrations/eas). It signs in with `EXPO_TOKEN`, else with the eas-cli login in `~/.expo/state.json`. On a CI runner, set `EXPO_TOKEN` or keep that login out of the runner's home. |
 | `mcr.microsoft.com` | Only from [`smol()`](/integrations/smol), to pull the Playwright image. The machines then have unrestricted network access, and reach every port on your computer's loopback through `host.smolvm.internal`, not only `hostPorts` ([Set up the provider](/integrations/smol#set-up-the-provider)). |
 | The package registry | When `e2e init` installs dependencies. |
 | The app under test | Every run, plus whatever the app itself loads. |
 
-There is no crash reporting, and no update check outside `e2e feedback`. `--ai-trace` writes a local file
-and doesn't upload it. Your test code, plugins, and app can make more
-connections.
+<Note>
+There is no crash reporting, and no update check outside `e2e feedback`.
+`--ai-trace` writes a local file and doesn't upload it. Your test code,
+plugins, and app can make more connections.
+</Note>
 
-## MCP and explore
+### MCP and explore
 
 | Command | Rule |
 | --- | --- |