@e2edev/e2e 0.12.0 → 0.14.0

package/skills/e2e/references/agent.md

@@ -1,8 +1,9 @@
 # Agent steps
 
-`agent` is a fixture like `screen`. Each call is one bounded invocation: a
-fresh redacted observation, a deadline, a model-call budget, and no shared
-transcript between calls. Deterministic tests never load a model.
+`agent` is a fixture like `screen` and the main way a test drives the app.
+Each call is one bounded invocation: a fresh redacted observation, a
+deadline, a model-call budget, and no shared transcript between calls. A
+test with no agent step never loads a model.
 
 ## Configure a model
 
@@ -15,18 +16,18 @@ import { gateway } from 'ai';
 
 export default {
   // 1. The built-in agent with an AI SDK model: gateway() from 'ai' is the Vercel AI Gateway and reads AI_GATEWAY_API_KEY.
-  agents: { default: createAgent({ model: gateway('openai/gpt-5.4-mini'), system: 'You are a thorough QA agent. Verify every outcome on screen.' }) },
+  agents: { default: createAgent({ model: gateway('openai/gpt-5.6-luna'), system: 'You are a thorough QA agent. Verify every outcome on screen.' }) },
 
   // 2. An options block: a model plus project vocabulary. openrouter() from '@openrouter/ai-sdk-provider' reads OPENROUTER_API_KEY.
   // agents: { default: { model: openrouter('anthropic/claude-sonnet-4.5'), context: 'A billing dashboard. Plans are Free, Team, and Pro.' } },
 
   // 3. A live AI SDK model instance for a provider called directly.
-  // agents: { default: createAgent({ model: openai('gpt-5.4-mini') }) },
+  // agents: { default: createAgent({ model: openai('gpt-5.6-luna') }) },
 } satisfies E2EConfig;
 ```
 
 ```bash
-AI_GATEWAY_API_KEY=... npx --no-install e2e run
+AI_GATEWAY_API_KEY=... npx e2e run
 ```
 
 - The model is always an AI SDK instance the config constructs; the runner
@@ -42,7 +43,6 @@ AI_GATEWAY_API_KEY=... npx --no-install e2e run
   lazily and fails without it.
 - `context` in the config and `agentContext` on a test or group add trusted
   project vocabulary to every prompt.
-- `visionModel` serves the calls that send pixels.
 - The model passed to `createAgent({ model })` is the one model for every
   `agent.*` call, `act` and the judgments alike.
   An agent `model` naming a different model is `INVALID_CONFIG`.
@@ -80,8 +80,15 @@ verdict.
 Options: `params` (the values the instruction refers to; a `Secret` is filled
 by the runner), `timeout` (default the test timeout), `maxSteps` (default 25
 actions), `maxModelCalls` (default 25). Per-call budgets can only lower the
-configured limits. `act` takes no `schema` and no `vision`: structured output
-is `extract({ schema })`, and `vision` belongs to the judgments.
+configured limits. `act` takes no `schema` and no `vision` option: structured
+output is `extract({ schema })`, and pixels reach an `act` step through the
+`screenshot` and `tap_at` tools the agent offers while no secret has been
+filled. `screenshot` attaches the viewport's pixels to the result and turns
+on pixel mode, where every action result carries a fresh screenshot; `tap_at`
+taps a point in the latest screenshot (a canvas shape, a map pin, an image
+region, a control in a system sheet), hit-tested against the tree first so a
+listed control is tapped by id. A screen with nothing to tap by id opens with
+a screenshot already attached.
 
 ## assert, waitFor, extract: one question
 
@@ -142,6 +149,14 @@ await expect(screen.getByRole('status')).toHaveText('Created "Atlas" on the Pro
 The check makes the test model-portable (the path may differ between models,
 the end state may not), and it is what lets the trace cache record the step.
 
+State the step writes off screen (a database row, an API read) can land after
+`act` returns; poll the read instead of sleeping:
+
+```ts
+await agent.act('create a test named "AI checkout regression"');
+await expect.poll(() => getTest(workspace).then((row) => row?.title), { timeout: 15_000 }).toBe('AI checkout regression');
+```
+
 ## What the model sees
 
 A redacted snapshot of the screen (roles, names, text, states), a summary of
@@ -150,8 +165,9 @@ every action result after it reports what changed, keyed by node ids that
 stay stable while an element exists, or the whole screen again when most of
 it changed, and is read after the action's effect landed. Never raw HTML,
 cookies, headers, environment
-values, or a `Secret`'s value; password fields arrive masked. Pixels only
-with `vision`, and only while no secret has been filled. Nothing the model
+values, or a `Secret`'s value; password fields arrive masked. Pixels reach a
+model only through `vision` on a judgment or the act loop's `screenshot` and
+pixel mode, masked, and only while no secret has been filled. Nothing the model
 returns runs as code or selectors: the runner validates and authorizes every
 tool call before it executes.
 
@@ -164,8 +180,9 @@ tool call before it executes.
 | `extract` | 2 | 30 s |
 | `waitFor` | up to `agent.maxModelCalls` (25) | 30 s |
 
-- Keep `agent.*` for steps whose path or wording varies; `expect` and
-  `screen` are free.
+- With the cache on, a passing `act` costs model calls once and replays on
+  later runs until the app changes. Budget for the runs where the UI moved,
+  not for every run.
 - Slow providers: raise the test `timeout` and `actionTimeout` (each
   observation and action inside a step is bounded by it) rather than reading
   latency as a defect. `STEP_TIMEOUT` and `STEP_BUDGET_EXHAUSTED` are test
@@ -178,14 +195,25 @@ tool call before it executes.
 Each passing `agent.act` records the actions it performed. The next run
 replays them with zero model calls and hands back to the live agent the
 moment the app no longer matches the recording, or when the recorded end
-state is not on screen after the replay.
+state is not on screen after the replay. Replayed actions run as a test's
+own steps do, without the agent's settle wait, so a replay is as fast as
+the deterministic equivalent.
 
 - On by default (`read-write`), `read-only` in CI, `cache: 'off'` in the
   config or `--no-cache` on a run to disable. Entries live in `.e2e/cache/`;
   deleting the directory only slows the next run.
-- An entry is written only after a later verification step passes
-  (`expect`, `locator.waitFor`, `agent.assert`, `agent.waitFor`). An `act`
-  nothing checks is never replayed.
+- An entry is written only after a later verification step passes: a
+  locator or engine `expect` matcher, `locator.waitFor`, `web.waitForURL`,
+  `agent.assert`, or `agent.waitFor`. A plain-value `expect`, `expect.poll`,
+  `agent.extract`, another `act`, or the attempt passing confirms nothing.
+  An `act` nothing checks is never replayed.
+- A replay needs the app on the path the step was recorded on, unless the
+  recording opens with a navigation. It re-finds each control by role, name,
+  test id, placeholder, and input purpose, and passes on its own only when
+  the recorded end path and the controls that appeared during the step are
+  back. Otherwise the agent takes over mid-step. The report's
+  `step.cache.reason` says why: `no-entry`, `wrong-context`,
+  `target-not-found`, `target-ambiguous`, `end-mismatch`, and so on.
 - A step that records no actions creates no entry and skips the cache's
   end-state observation.
 - `e2e init` gitignores `.e2e/cache/`; committing entries is opt-in. Remove
@@ -197,8 +225,8 @@ state is not on screen after the replay.
 ## Inspect what the model did
 
 ```bash
-npx --no-install e2e run tests/checkout.e2e.ts --debug      # step table, transcripts as artifacts
-npx --no-install e2e run tests/checkout.e2e.ts --ai-trace   # writes .e2e/ai-trace.json
+npx e2e run tests/checkout.e2e.ts --debug      # step table, transcripts as artifacts
+npx e2e run tests/checkout.e2e.ts --ai-trace   # writes .e2e/ai-trace.json
 npx unbox-ai runs .e2e/ai-trace.json                        # one line per agent step
 npx unbox-ai summary .e2e/ai-trace.json --run 0             # turns, tokens, tool calls of one step
 ```
@@ -211,6 +239,32 @@ The trace replaces inline bytes and base64 in SDK image and file message
 parts with decoded byte counts. URLs and text remain; encoded strings in
 arbitrary tool result JSON or other fields are preserved.
 
+## Make the agent yours
+
+The agent in the config is a starting point. The best agent for an app is
+the one that knows its screens, and that comes from iterating on it:
+
+1. **The goal.** A failed step usually means the goal named something the
+   screen does not. Reword it with the labels on screen. Check the step's
+   transcript with `--debug` to see what the model saw and tried.
+2. **`context`.** Vocabulary every step needs: what the plans are called,
+   what a "workspace" is, which tab holds billing. Set it once on the agent
+   or per test with `agentContext`, not in every instruction.
+3. **`system` on `createAgent`.** How the agent works: how carefully it
+   verifies, what it never does, how it treats a modal. A UX reviewer, a
+   cautious QA persona, and a fast smoke agent are three `system` prompts on
+   the same model.
+4. **Tools.** A test API the agent may call mid-flow (seed a cart, mint a
+   coupon) via `createAgent({ tools })`; see below.
+5. **The model and its options.** `providerOptions` for reasoning effort,
+   or a different model for one persona. `npx e2e run --agent <name>` runs
+   the suite as any configured agent, so two candidates can be compared on
+   the same tests; every result records which agent ran it.
+
+Personas are agents by name under `agents`, pinned with `{ agent }` on a
+test or block, or swept with `--agent buyer,admin`. The trace cache records
+per agent step, so a specialised agent gets the same replay benefit.
+
 ## Beyond the built-in agent
 
 - `createAgent({ tools: { seedCart } })` adds AI SDK tools wrapped with
@@ -226,8 +280,10 @@ Full reference: https://e2e.mintlify.app/agents
 
 ## In CI
 
-Deterministic tests gate merges; agentic tests are opt-in. Keep them in a
-separate config (`e2e.agent.config.ts` with its own `tests` glob and a larger
-`timeout`), run them on a schedule or `workflow_dispatch`, and pass the key
-the config's model reads (`env: { AI_GATEWAY_API_KEY }` for `gateway()`) from
-secrets.
+One suite, one job, on every pull request, agent steps included. Pass the
+key the config's model reads (`env: { AI_GATEWAY_API_KEY }` for `gateway()`)
+from secrets. Commit `.e2e/cache/` so CI replays recorded steps with no
+model call and consults the model only where the app changed; keep every
+`act` followed by a check so the recording is trusted. CI retries once and
+reports a pass-on-retry as flaky, so the report keeps naming the goals that
+need tightening.

package/skills/e2e/references/debugging.md

@@ -28,7 +28,7 @@ jq '.run.results[] | select(.status != "passed") | .attempts[-1]
 | `CONFIG_NOT_FOUND`, `CONFIG_AMBIGUOUS` | Wrong `--config` path; both `.ts` and `.mts` present | Fix the path; keep one config file |
 | `NO_TESTS` | The glob or a positional matched nothing | The message names each positional that matched nothing. Check `tests` in the config and the `.e2e.ts` suffix |
 | `COLLECTION_ERROR` | `async` describe body, `test.setup` inside `describe`, an option forbidden in a serial group, registration outside collection | Rework the structure per `writing-tests` |
-| `APP_UNREACHABLE` | `command` never answered `readyUrl` within `startupTimeout`; a service exited non-zero | Set `command.log` and read it; check the port and `url`; pass the env the app needs through `command.env`; raise `startupTimeout` |
+| `APP_UNREACHABLE` | `command` never answered `readyUrl` within `startupTimeout`; a service exited non-zero | Read the last log lines quoted under the error; set `command.log` if it says to; check the port and `url`; pass the env the app needs through `command.env`; raise `startupTimeout` |
 | `APP_ALREADY_RUNNING` | Something already serves `url` when the runner wanted to start `command` | Stop it, or set `reuseExisting: true` for local runs |
 | `APP_URL_REQUIRED` | A navigation on an engine without `url` | Add `url` to `playwright({ ... })` |
 | `LOCATOR_NOT_FOUND` | Wrong role or name, text not exact, element off screen or inside an iframe, page not open | Read the markup for the accessible name; try `exact: false` or a RegExp; `web.frameLocator` for iframes; `app.open()` first; `--headed` to look |
@@ -36,16 +36,17 @@ jq '.run.results[] | select(.status != "passed") | .attempts[-1]
 | `ASSERTION_FAILED` | The expectation is wrong, or the state settles later than 5 s; for `agent.assert`, the judgment was false (explanation in the report) | Compare with the actual text in the report or screenshot; `{ timeout }` on the matcher; rewrite the question |
 | `ACTION_FAILED` | Element not actionable (covered, disabled, detached) or an operation timed out | Wait on the right condition with `expect` first; close overlays; check `actionTimeout` |
 | `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` for slow flows and agent steps |
-| `MODEL_UNAVAILABLE` | No model: neither `createAgent({ model })` nor `agent.model` holds an AI SDK instance. Reported once for the run under `run.errors`; the run stops | Construct one in the config, e.g. `gateway('openai/gpt-5.4-mini')` from `ai`, and export the key its provider reads (`AI_GATEWAY_API_KEY`) |
+| `MODEL_UNAVAILABLE` | No model: neither `createAgent({ model })` nor `agent.model` holds an AI SDK instance. Reported once for the run under `run.errors`; the run stops | Construct one in the config, e.g. `gateway('openai/gpt-5.6-luna')` from `ai`, and export the key its provider reads (`AI_GATEWAY_API_KEY`) |
 | `MODEL_PROVIDER_FAILED` | Network, 5xx, rate limit, or no credits after the transport retries | Check the key and the quota; retry; exit code 3 |
 | `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | The goal was too big or ambiguous, or the provider slow | Split the goal, use on-screen wording, add `context`, raise `timeout` and `actionTimeout`, `--debug` to read the transcript |
 | `CONTEXT_OVERFLOW` | The screen plus the step's history did not fit the model's context window, even after the loop shrank the history and retried once | Lower `agent.maxObservationBytes`, split the step, or pick a model with a larger window |
-| `POLICY_DENIED` | Navigation outside `allowedOrigins`; a `Secret` given to a sink that is not a password field; reading a secure field | Add the origin to `allowedOrigins`; fill secrets only into password inputs; assert the outcome instead of the value |
+| `POLICY_DENIED` | Navigation outside `allowedOrigins`; a password `Secret` given to a sink that is not a password field; reading a secure field; `app.screenshot()` after a secret fill | Add the origin to `allowedOrigins`; fill passwords only into password inputs; assert the outcome instead of the value; capture screenshots before filling secrets |
 | `UNSUPPORTED_CAPABILITY` | A fixture the engine does not contribute (`web` on a device), `schema` or `vision` on `act`, an action the surface lacks | Declare `requires: ['web']`; drop the option; use a supported action |
 | `SESSION_UNAVAILABLE`, `SESSION_CONTRACT` | `session: 'x'` with no setup saving `x`; a setup that did not save every declared name | Add or fix the `test.setup` |
 | `ONLY_IN_CI` | `test.only` reached CI | Remove it |
 | `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download or launch failed | `npx playwright install chromium --with-deps`; raise `launchTimeout` on slow machines |
 | `AUTH_CREDENTIAL_UNAVAILABLE` | `credentials.user('x')` for an undeclared name | Add it to `config.credentials` |
+| `SECRET_UNAVAILABLE` | `secrets.get('x')` for an undeclared name | Add it to `config.secrets` or set `E2E_SECRET_X` |
 
 ## Tools
 
@@ -58,8 +59,8 @@ jq '.run.results[] | select(.status != "passed") | .attempts[-1]
 | `--ai-trace`, then `npx unbox-ai runs .e2e/ai-trace.json` | See exactly what the model was shown and called |
 | `--video` | Watch the failed attempt; `step.startedAt` minus the video artifact's `startedAt` is the step's offset into it |
 | `command.log: '.e2e/logs/app.log'` | Read the app's own output when it never becomes ready or errors mid-test |
-| `await app.screenshot('before-submit')` | Attach evidence at a chosen point |
-| `CI=1 npx --no-install e2e run` | Reproduce CI-only behaviour: `ONLY_IN_CI`, read-only cache, `reuseExisting` ignored |
+| `await app.screenshot('before-submit')` | Attach evidence before any secret is filled; later calls fail with `POLICY_DENIED` |
+| `CI=1 npx e2e run` | Reproduce CI-only behaviour: `ONLY_IN_CI`, read-only cache, `reuseExisting` ignored |
 
 ## Flaky tests
 

package/skills/e2e/references/explore.md

@@ -7,10 +7,10 @@ test. It needs a config with a target and an agent that holds a model,
 nothing else.
 
 ```bash
-npx --no-install e2e explore                                         # goal: "Explore the app and find bugs"
-npx --no-install e2e explore 'Explore checkout like a first-time buyer and report anything off'
-npx --no-install e2e explore --target web --max-steps 4 --headed
-npx --no-install e2e explore 'Hunt for broken forms' --video
+npx e2e explore   # goal: "Explore the app and find bugs"
+npx e2e explore 'Explore checkout like a first-time buyer and report anything off'
+npx e2e explore --target web --max-steps 4 --headed
+npx e2e explore 'Hunt for broken forms' --video
 ```
 
 ## What a run does
@@ -53,9 +53,15 @@ Exit code `0`: steps ran and no `issue` was reported (warnings allowed). Exit
 code `1`: at least one `issue`, or no step ran and nothing was found (the run
 is `blocked`). `2` and `3` as for `run`.
 
-The terminal prints each step live and, under the summary, `Explored`,
-`Findings`, one row per finding sorted by severity, and `Assessment`.
-`.e2e/report.json` has the record under `run.explore`:
+The terminal shows the exploration step by step: each step by its title with
+its duration, actions, and findings, and each finding the moment it is
+reported as `⚑ high issue  Title (/path)`. At the end a `Findings` section
+lists every finding, issues first and the most severe first, each with where
+it was seen, its screenshot path, what was expected against what the screen
+showed, and the steps that reach it; then the `Assessment` and the summary
+(`Findings`, `Steps`, `AI`, `Duration`, `Report`). Severity words: critical 5,
+high 4, medium 3, low 2, trivial 1. `.e2e/report.json` has the record under
+`run.explore`:
 
 ```json
 {

package/skills/e2e/references/mcp.md

@@ -12,14 +12,14 @@ The server ships with `@e2edev/e2e`. `e2e init` offers to register it; by
 hand:
 
 ```bash
-claude mcp add e2e -- npx --no-install e2e mcp          # Claude Code
+claude mcp add e2e -- npx e2e mcp   # Claude Code
 ```
 
 Or declare it in the client's project config (`.mcp.json` for Claude Code,
 `.cursor/mcp.json` for Cursor, `.vscode/mcp.json` for VS Code):
 
 ```json
-{ "mcpServers": { "e2e": { "command": "npx", "args": ["--no-install", "e2e", "mcp"] } } }
+{ "mcpServers": { "e2e": { "command": "npx", "args": ["e2e", "mcp"] } } }
 ```
 
 Flags: `--config <path>` names the default config file, `--target <name>`
@@ -45,7 +45,7 @@ The catalog, per session:
 | --- | --- |
 | `observe` | A fresh observation: one node per line as `#id role "name" ...`, plus the current path. |
 | `tap`, `type`, `press`, `select`, `scroll`, `navigate` | The grammar verbs, exactly as the testing agent gets them. Each reports what changed on screen; `observe` shows the whole screen. A verb the engine cannot honor is not listed and fails with `UNSUPPORTED_CAPABILITY`. |
-| `type_secret` | Fills a configured credential's password into a secure field by credential name; the plaintext never reaches the agent. Listed when the config declares `credentials`. |
+| `type_secret` | Fills a configured secret by name: a credential's password into a password field, a `secrets` entry into any editable input; the plaintext never reaches the agent. Listed when the config declares `credentials` or `secrets`. |
 | `locate` | Tries a semantic locator (`role` + `name`, `text`, `label`, `placeholder`, `testId`, `exact`) and returns how many nodes match, which, and the `screen.*` call to write. |
 | `screenshot` | The masked pixels as an image, withheld once a secret was filled in the session. |
 | Project tools | Every `defineTool` passed to `createAgent({ tools })` that applies to the target's platform, under its own name; an engine pack such as `agentDeviceTools` adds `open_app`, `swipe`, `type_text`, `alert`. |
@@ -65,7 +65,7 @@ Resources: `e2e://guide` and `e2e://guide/<topic>` hold this skill.
    `LOCATOR_NOT_FOUND` or `LOCATOR_AMBIGUOUS`.
 3. Write `tests/<feature>.e2e.ts` (topic `writing-tests`). Deterministic steps
    where you saw exact names; `agent.act` where the flow varies.
-4. Run it from the shell: `npx --no-install e2e run tests/<feature>.e2e.ts`,
+4. Run it from the shell: `npx e2e run tests/<feature>.e2e.ts`,
    read the failure (topic `debugging`), fix, repeat.
 5. `close_session` when you are done exploring; an idle session closes on its
    own after 30 minutes and never outlives 4 hours. To look at another

package/skills/e2e/references/running.md

@@ -3,28 +3,28 @@
 ## Commands
 
 ```bash
-npx --no-install e2e run [files...] [options]   # run tests
-npx --no-install e2e explore [goal] [options]   # explore the app toward a goal, no test file (see the explore topic)
-npx --no-install e2e list [files...] [options]  # print what run would select, without running
-npx --no-install e2e init [--yes]               # scaffold a project, refresh the agent skill
-npx --no-install e2e guide [topic]              # print this skill: setup, writing-tests, agent, running, explore, debugging, mcp
-npx --no-install e2e cache ls|clear|stats       # read or empty the trace cache
-npx --no-install e2e mcp [--target <name>]      # serve the project to a coding agent over MCP (topic mcp)
-npx --no-install e2e telemetry [disable|enable] # anonymous usage telemetry: status, or the switch
+npx e2e run [files...] [options]   # run tests
+npx e2e explore [goal] [options]   # explore the app toward a goal, no test file (see the explore topic)
+npx e2e list [files...] [options]  # print what run would select, without running
+npx e2e init [--yes]               # scaffold a project, refresh the agent skill
+npx e2e guide [topic]              # print this skill: setup, writing-tests, agent, running, explore, debugging, mcp
+npx e2e cache ls|clear|stats       # read or empty the trace cache
+npx e2e mcp [--target <name>]      # serve the project to a coding agent over MCP (topic mcp)
+npx e2e telemetry [disable|enable] # anonymous usage telemetry: status, or the switch
 ```
 
 `run` flags:
 
 | Flag | Effect |
 | --- | --- |
-| `[files...]` | Files, directories, or quoted globs relative to the project root. They narrow the config `tests` glob, never bypass it. |
+| `[files...]` | Files, directories, or quoted globs relative to the project root, or a bare file name (`signup.e2e.ts`, `signup`, `agent/signup.e2e.ts` all select `tests/agent/signup.e2e.ts`). They narrow the config `tests` glob, never bypass it. |
 | `--config <path>` | Explicit config file. Default: `e2e.config.ts` or `.mts` found upward from the working directory. |
 | `--target <ids>` | Comma-separated target names. Only selected targets start app commands and services; unknown names fail before startup. |
 | `--tag <tag>` | Repeatable tag filter; `--tag-mode all` requires every tag. |
 | `--headed` | Visible browser or simulator when the engine supports it. |
 | `--agent <names>` | Run unpinned tests as other configured agents (`agents.<name>`), comma-separated or repeated; several names run each such test once per agent. Default is `agents.default`. |
 | `--workers <n>`, `--retries <n>` | Override the resolved values. |
-| `--reporter <ids>` | `list`, `json`, `junit`, comma-separated. `json` cannot combine with `list`. |
+| `--reporter <ids>` | `list`, `json`, `junit`, `markdown`, comma-separated. `json` cannot combine with `list`. |
 | `--artifacts <dir>` | Artifact root, default `.e2e/artifacts`. |
 | `--no-cache` | Run with the trace cache off. |
 | `--pass-with-no-tests` | Exit 0 when nothing matches instead of `NO_TESTS`. |
@@ -33,10 +33,11 @@ npx --no-install e2e telemetry [disable|enable] # anonymous usage telemetry: sta
 | `--video` | Record every attempt (WebM on a browser engine, MP4 on a device engine) under its artifact directory; the failure recap names the file. Fails with `UNSUPPORTED_ARTIFACT` when the engine cannot record. |
 
 ```bash
-npx --no-install e2e run tests/signup.e2e.ts
-npx --no-install e2e run tests/agent --tag smoke
-npx --no-install e2e run 'tests/**/*.smoke.e2e.ts' --target chromium --workers 1 --retries 0
-CI=1 npx --no-install e2e run   # reproduce the CI defaults locally
+npx e2e run tests/signup.e2e.ts
+npx e2e run signup.e2e.ts   # the same file by name, from any directory the config globs cover
+npx e2e run tests/agent --tag smoke
+npx e2e run 'tests/**/*.smoke.e2e.ts' --target chromium --workers 1 --retries 0
+CI=1 npx e2e run            # reproduce the CI defaults locally
 ```
 
 `list` takes the same files and the selection flags (`--config`, `--target`,
@@ -46,12 +47,15 @@ app, an engine, or a worker. `--reporter json` prints `{ "pairs": [...] }`.
 Use it to check a filter before a run.
 
 ```bash
-npx --no-install e2e list --tag smoke
-npx --no-install e2e list tests/signup.e2e.ts --reporter json
+npx e2e list --tag smoke
+npx e2e list tests/signup.e2e.ts --reporter json
 ```
 
 A `package.json` script keeps it short: `"test:e2e": "e2e run"`, then
-`pnpm test:e2e tests/signup.e2e.ts`.
+`pnpm test:e2e tests/signup.e2e.ts`. pnpm forwards a `--` separator literally,
+so `pnpm test:e2e -- --headed` reaches e2e as `run -- --headed` and is
+rejected with exit 2 rather than run headless; write `pnpm test:e2e --headed`
+or `pnpm exec e2e run --headed` instead.
 
 ## The trace cache
 
@@ -80,6 +84,12 @@ recorded flow is stale — `--no-cache` only skips the cache for one run.
   `artifacts[]`, and `error`.
 - `junit`: `.e2e/junit.xml` beside the report, for CI test summaries.
   Combine it with the terminal output: `--reporter list,junit`.
+- `markdown`: `.e2e/summary.md` beside the report: the counts and what the
+  run spent, a block per failed or flaky test with its error, the step it went
+  wrong at, the agent's explanation, and evidence paths, a row per file, every
+  test folded; or an exploration's findings and assessment. Paste it into a
+  pull request or a handoff rather than retelling the result:
+  `--reporter list,markdown`.
 - `json`: the report document on stdout.
 - `github()` from `@e2edev/github`: on GitHub Actions, one pull request comment per
   run (edited on rerun) and the job summary; needs `pull-requests: write` and
@@ -133,7 +143,7 @@ jobs:
           cache: pnpm
       - run: pnpm install --frozen-lockfile
       - run: npx playwright install chromium --with-deps
-      - run: npx --no-install e2e run --reporter list,junit
+      - run: npx e2e run --reporter list,junit
         env:
           E2E_USER_ADMIN_USERNAME: ${{ secrets.E2E_USER_ADMIN_USERNAME }}
           E2E_USER_ADMIN_PASSWORD: ${{ secrets.E2E_USER_ADMIN_PASSWORD }}
@@ -156,6 +166,7 @@ jobs:
   launch timeout.
 - Start the app through the engine's `command`; the runner tears it down on
   every exit path.
-- Agentic suites: a separate config, run on `schedule` or
-  `workflow_dispatch`, the key the config's model reads (`AI_GATEWAY_API_KEY`
-  for `gateway()` from `ai`) as a secret, never a required check.
+- Agent steps run in the same job as everything else. Pass the key the
+  config's model reads (`AI_GATEWAY_API_KEY` for `gateway()` from `ai`) as
+  a secret in the run step's `env`, and commit `.e2e/cache/` so recorded
+  steps replay in CI with no model call.

package/skills/e2e/references/setup.md

@@ -7,9 +7,17 @@
   whatever the nearest `package.json` `type` says, so a CommonJS package (a
   Next.js app, for instance) needs no change. Write them with `import`, never
   `require` or `module.exports`.
-- For browser tests, `@e2edev/playwright`. Missing browsers download when the
-  engine first boots. In CI install them up front:
-  `npx playwright install chromium --with-deps`.
+- For browser tests, `@e2edev/playwright` plus `playwright` (`>=1.63.0 <2`),
+  a peer dependency the engine does not install itself: an app that already
+  depends on Playwright keeps its version and its browser cache. A version
+  outside the range may be rejected by the package manager as an unmet peer
+  (npm's `ERESOLVE`); upgrade `playwright` within the range.
+  Missing browsers download when the engine first boots. In CI install them
+  up front: `npx playwright install chromium --with-deps`.
+- For mobile tests, `@e2edev/agent-device` plus `agent-device` (`0.21.x`),
+  likewise a peer dependency: a project that already drives devices with the
+  agent-device CLI keeps its version. The range pins the tested minor because
+  agent-device minors break; it moves with each engine release.
 
 ## Scaffold
 
@@ -18,8 +26,8 @@ npx @e2edev/e2e@beta init       # npm
 pnpm dlx @e2edev/e2e@beta init  # pnpm
 ```
 
-When `@e2edev/e2e` is already installed, run `npx --no-install e2e init`
-instead, so the installed version scaffolds.
+When `@e2edev/e2e` is already installed, run `npx e2e init` instead, so the
+installed version scaffolds.
 
 The wizard asks for the engine (Web with Playwright by default; Mobile with
 agent-device and None are the alternatives), which model gateway agent steps
@@ -42,7 +50,7 @@ project.
 Without the wizard:
 
 ```bash
-npm install --save-dev @e2edev/e2e@beta @e2edev/playwright@beta ai@^7
+npm install --save-dev @e2edev/e2e@beta @e2edev/playwright@beta playwright ai@^7
 ```
 
 `ai` (the Vercel AI SDK, v7) is only needed for `agent.*` steps.
@@ -69,10 +77,10 @@ export default {
       }),
     },
   ],
-  // Only for agent.* steps. The model is an AI SDK instance; gateway() from 'ai' reads AI_GATEWAY_API_KEY.
+  // The model behind every agent.* step: an AI SDK instance; gateway() from 'ai' reads AI_GATEWAY_API_KEY.
   agents: {
     default: createAgent({
-      model: gateway('openai/gpt-5.4-mini'),
+      model: gateway('openai/gpt-5.6-luna'),
       system: 'You are a thorough QA agent. Verify every outcome on screen.',
     }),
   },
@@ -91,10 +99,11 @@ export default {
 | `assertionTimeout` | `5000` | `expect` polling window. |
 | `retries` | `0`, `1` in CI | 0 to 10. |
 | `workers` | half the cores, `1` in CI | Test files run in parallel across workers, at most the `workers` the engine declares per target (a device target: one per device). |
-| `reporters` | `['list']` | `list`, `json`, `junit`, and reporter objects (`{ name, onEvent?, onRunFinished? }`) that receive the finished run. `json` excludes `list`; `--reporter` keeps the objects. |
+| `reporters` | `['list']` | `list`, `json`, `junit`, `markdown`, and reporter objects (`{ name, onEvent?, onRunFinished? }`) that receive the finished run. `json` excludes `list`; `--reporter` keeps the objects. |
 | `cache` | `'read-write'`, `'read-only'` in CI | The trace cache for `agent.act`; `'off'` disables it. |
-| `agents` | `{ default: built-in }` | Agents by name. `default` is what tests run with; `e2e run --agent <name>` runs with another. Each entry is `createAgent(...)`, an options block `{ model, context, visionModel, maxSteps, maxModelCalls, vision, providerOptions }`, or a custom `StepExecutor`. `model` is an AI SDK instance; without one anywhere, acquiring `agent` is `MODEL_UNAVAILABLE`. |
+| `agents` | `{ default: built-in }` | Agents by name. `default` is what tests run with; `e2e run --agent <name>` runs with another. Each entry is `createAgent(...)`, an options block `{ model, context, maxSteps, maxModelCalls, vision, providerOptions }`, or a custom `StepExecutor`. `model` is an AI SDK instance; without one anywhere, acquiring `agent` is `MODEL_UNAVAILABLE`. |
 | `credentials` | `{}` | Named `{ username, password, allowedOrigins? }` entries; `password` may be a function returning the value. |
+| `secrets` | `{}` | Named values the model never sees (API keys, tokens): a string, a function returning the value, or `{ value, allowedOrigins? }`. A name cannot also be a credential. |
 | `screen.testIdAttribute` | `'data-testid'` | Attribute read by `getByTestId`. |
 | `artifacts` | `['screenshot', 'trace']` | Kinds to keep (`screenshot`, `trace`, and the opt-in `video`), or `{ kinds, store, video }`; `video: { retain: 'on-failure' }` keeps only the recordings of attempts that did not pass. |
 | `projectId` | the package name | Report and cache identity. |
@@ -106,9 +115,9 @@ process, and identity. `playwright()` accepts:
 
 | Option | Meaning |
 | --- | --- |
-| `url` | Base URL for `app.open()` and relative navigation. A missing scheme becomes `https://`, or `http://` for a loopback host. Required once a test navigates. |
-| `command` | The process that serves `url`. See below. |
-| `readyUrl` | Readiness probe when it differs from `url`. |
+| `url` | Base URL for `app.open()` and relative navigation. A missing scheme becomes `https://`, or `http://` for a loopback host. Required once a test navigates. Port `0` on `127.0.0.1` or `[::1]` asks the run for a free port. |
+| `command` | The process that serves `url`. `{port}` in `args` and `env` expands to the port of `url`. See below. |
+| `readyUrl` | Readiness probe when it differs from `url`. `{port}` expands here too. |
 | `services` | Dependency processes started before `command`, in order. |
 | `allowedOrigins` | Origins tests and the agent may navigate to. Default: the origin of `url`. |
 | `environment` | `'test'`, `'staging'`, `'production'`. Inferred from the host; a label for the report and the cache key. |
@@ -181,6 +190,15 @@ How it behaves:
   then runs their `teardown` commands in reverse.
 - `executable` is resolved on `PATH` and never shell-interpreted. Point it at
   the server itself rather than at a wrapper script.
+- `url: 'http://127.0.0.1:0'` (or `[::1]:0`, never `localhost:0`) asks the
+  run for a free port, so two checkouts can run at once. The command must take
+  it through `{port}` in `args` or `env`; the token also expands in `readyUrl`
+  and the services:
+  `command: { executable: 'pnpm', args: ['dev', '--port', '{port}'], env: { PORT: '{port}' } }`.
+  Tests read the allocated URL from `app.baseUrl`; the cache identity keeps
+  the declared `:0`. Services keep their own ports. A port another process
+  grabs between allocation and spawn fails the start with `APP_UNREACHABLE`;
+  rerun.
 
 To test an app started elsewhere, point `url` at it and start it yourself,
 or read the address from the environment:
@@ -193,6 +211,7 @@ or read the address from the environment:
 | --- | --- |
 | `AI_GATEWAY_API_KEY`, `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, ... | Read by the provider package the config constructs the model with (`gateway()` from `ai`, `openrouter()`, `openai()`), not by the runner. The runner reads no model variable; the model is always an AI SDK instance in the config. |
 | `E2E_USER_<NAME>_USERNAME`, `E2E_USER_<NAME>_PASSWORD` | Override `credentials.<name>`. `<NAME>` is the credential name uppercased, other characters as `_`. |
+| `E2E_SECRET_<NAME>` | Overrides the value of `secrets.<name>`, same uppercasing rule. |
 | `CI` | Turns on CI defaults: `retries: 1`, `workers: 1`, `test.only` rejected, cache read-only, `reuseExisting` ignored. |
 | `E2E_TELEMETRY_DISABLED`, `DO_NOT_TRACK` | Turn anonymous usage telemetry off; `e2e telemetry disable` saves the same choice. `E2E_TELEMETRY_DEBUG=1` prints each event instead of sending it. |
 
@@ -227,6 +246,11 @@ export default {
   single `device` runs one worker whatever `workers` says; a list,
   `device: ['iPhone 17', 'iPhone 17 Pro']`, is an explicit pool. Devices boot
   in `prepare`, before the run's clock starts.
+- A test's steps never wait for the screen to settle; `expect` verifies the
+  outcome. Only a control that appeared or moved with the previous action
+  waits out the `transition` budget (default 500 ms) before it is acted on.
+  Agent actions settle for `settle` ms (default 150) before the agent
+  observes again; `settle: false` skips that wait.
 - Cancelled device commands keep running; the next attempt waits for them.
   Raw screenshot files are removed when capture finishes, including after
   cancellation.
@@ -244,8 +268,10 @@ export default {
 
 ## Done when
 
-- `npx --no-install e2e run tests/example.e2e.ts` passes against the app.
+- `npx e2e run tests/example.e2e.ts` passes against the app.
 - `package.json` has a script such as `"test:e2e": "e2e run"`.
 - `.gitignore` lists the `.e2e/` outputs (init adds them). Committing
-  `.e2e/cache/` is opt-in: remove that line to share `agent.act` replays.
-- CI runs the deterministic suite on pull requests; see `running`.
+  `.e2e/cache/` is opt-in: remove that line so CI and teammates replay
+  `agent.act` steps instead of re-running the model.
+- CI runs the whole suite on pull requests, agent steps included; see
+  `running`.