@e2edev/e2e 0.5.0 → 0.6.0

package/skills/e2e/references/agent.md

@@ -0,0 +1,210 @@
+# Agent steps
+
+`agent` is a fixture like `screen`. Each call is one bounded invocation: a
+fresh redacted observation, a deadline, a model-call budget, and no shared
+transcript between calls. Deterministic tests never load a model.
+
+## Configure a model
+
+There is no default model. Pick one of three shapes:
+
+```ts
+// e2e.config.ts
+import { createAgent } from '@e2edev/e2e/agent';
+
+export default {
+  // 1. The built-in agent; the model comes from E2E_MODEL at run time.
+  agent: createAgent({ system: 'You are a thorough QA agent. Verify every outcome on screen.' }),
+
+  // 2. An options block: a gateway model string plus project vocabulary.
+  // agent: { model: 'anthropic/claude-sonnet-4.5', context: 'A billing dashboard. Plans are Free, Team, and Pro.' },
+
+  // 3. A live AI SDK model instance for a provider called directly.
+  // agent: createAgent({ model: openai('gpt-5.4-mini') }),
+} satisfies E2EConfig;
+```
+
+```bash
+E2E_MODEL=anthropic/claude-sonnet-4.5 E2E_MODEL_API_KEY=... npx --no-install e2e run
+```
+
+- A `provider/model-id` string is routed through the Vercel AI Gateway; one
+  `E2E_MODEL_API_KEY` (or `AI_GATEWAY_API_KEY`) reaches every provider.
+- With no `agent` key at all, the built-in agent still runs and takes its
+  model from `E2E_MODEL`. Use `createAgent` for a `system` prompt, tools, or
+  a pinned model.
+- `ai@^7` must be installed for any `agent.*` step; the runner loads it
+  lazily and fails without it.
+- `context` in the config and `agentContext` on a test or group add trusted
+  project vocabulary to every prompt.
+- `visionModel` (or `E2E_VISION_MODEL`) serves the calls that send pixels.
+- Missing model or key: the `agent` fixture fails with `MODEL_UNAVAILABLE`
+  (exit 2) at first use.
+
+## act: one goal
+
+```ts
+import { credentials } from '@e2edev/e2e';
+
+await agent.act('add a todo named "Buy milk" and mark it done');
+await agent.act('invite {email} as an editor', { email: 'ada@example.test' });
+
+const member = credentials.user('member');
+await agent.act('sign in with the given credentials', {
+  username: member.username,
+  password: member.password, // a Secret: the model sees its name, the runner fills the field
+});
+```
+
+`act(instruction, params?, options?)` plans and performs a multi-action flow
+and ends in a verdict. Passed returns `{ ok: true }`. Failed or blocked
+throws an `AgentError` whose `code` says why: `ACTION_FAILED` for a plain
+failure, `STEP_BUDGET_EXHAUSTED` or `STEP_TIMEOUT` when the budget or the
+clock ran out, and a blocked code (`AUTH_CREDENTIAL_UNAVAILABLE`,
+`ENVIRONMENT_UNAVAILABLE`, `SEED_DATA_MISSING`, `TEST_SETUP_FAILED`,
+`AUTOMATION_UNSUPPORTED`) when something outside the product prevented a
+verdict.
+
+Options: `timeout` (default the test timeout), `maxSteps` (default 25
+actions), `maxModelCalls` (default 25). Per-call values can only lower the
+configured limits. `schema` and `vision` on `act` are not implemented and
+reject with `UNSUPPORTED_CAPABILITY`.
+
+## assert, waitFor, extract: one question
+
+```ts
+import { z } from 'zod';
+
+await agent.assert('the dashboard shows a trial badge'); // one look, one judgment
+
+await agent.waitFor('the export finished and a download link appeared', { // polls
+  intervalMs: 500,
+  timeout: 120_000,
+});
+
+const data = await agent.extract('every todo title and how many remain', { // structured output
+  schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }),
+});
+expect(data.titles).toContain('Buy milk');
+```
+
+- `assert` does not poll. A false judgment is `ASSERTION_FAILED` with the
+  model's explanation and a screenshot in the report. Malformed output gets
+  one repair round, then `MODEL_OUTPUT_INVALID`.
+- `waitFor` observes every `intervalMs` (default 3 s) and spends a judgment
+  only when the screen changed; `STEP_TIMEOUT` after `timeout` (default
+  30 s).
+- `extract` accepts any Standard Schema validator (zod works). Invalid output
+  gets one repair round, then `MODEL_OUTPUT_INVALID`.
+- Judgments are never cached and always read a fresh observation.
+
+`vision` on a judgment controls the evidence: `false` (default) the semantic
+tree; `true` the tree plus a masked screenshot; `'only'` the screenshot
+alone. Use `'only'` for a question about what the screen presents (an
+overlay, a broken layout, a chart), because the tree would otherwise answer
+first. Pixels show the viewport only and are withheld once a secret was
+filled in the attempt.
+
+## Write instructions the model can execute
+
+- One goal per `act`. The order of goals is the test's; the path inside a
+  goal is the model's.
+- Use the words on screen: `'open the Billing tab and choose the Pro plan'`,
+  not `'upgrade'` when no control says so.
+- Values go in params, never in the sentence:
+  `act('rename the project to {name}', { name })`. The runner does not
+  expand `{name}`; the model receives the instruction as written plus the
+  params as a separate block and reads the value from there.
+- Give vocabulary once, in `agent.context` or `agentContext`, instead of
+  repeating it in every instruction.
+- Do not describe mechanics the runner already handles: waiting, scrolling
+  into view, retries.
+- Pin the outcome of every `act` deterministically, right after it:
+
+```ts
+await agent.act('create a workspace named "Atlas" on the Pro plan');
+await expect(screen.getByRole('status')).toHaveText('Created "Atlas" on the Pro plan');
+```
+
+The check makes the test model-portable (the path may differ between models,
+the end state may not), and it is what lets the trace cache record the step.
+
+## What the model sees
+
+A redacted snapshot of the screen (roles, names, text, states), a summary of
+prior steps, and your context. Never raw HTML, cookies, headers, environment
+values, or a `Secret`'s value; password fields arrive masked. Pixels only
+with `vision`, and only while no secret has been filled. Nothing the model
+returns runs as code or selectors: the runner validates and authorizes every
+tool call before it executes.
+
+## Budgets and cost
+
+| Call | Model calls | Default timeout |
+| --- | ---: | --- |
+| `act` | up to `agent.maxModelCalls` (25) | the test `timeout`, 120 s |
+| `assert` | 2 | 30 s |
+| `extract` | 2 | 30 s |
+| `waitFor` | up to `agent.maxModelCalls` (25) | 30 s |
+
+- Keep `agent.*` for steps whose path or wording varies; `expect` and
+  `screen` are free.
+- Slow providers: raise the test `timeout` and `actionTimeout` (each
+  observation and action inside a step is bounded by it) rather than reading
+  latency as a defect. `STEP_TIMEOUT` and `STEP_BUDGET_EXHAUSTED` are test
+  failures: scope the goal smaller or split it.
+- `--debug` prints a per-step table (duration, model calls, tokens, cost)
+  after the run and saves each step's transcript as an artifact.
+
+## The trace cache
+
+Each passing `agent.act` records the actions it performed. The next run
+replays them with zero model calls and hands back to the live agent the
+moment the app no longer matches the recording, or when the recorded end
+state is not on screen after the replay.
+
+- On by default (`read-write`), `read-only` in CI, `cache: 'off'` in the
+  config or `--no-cache` on a run to disable. Entries live in `.e2e/cache/`;
+  deleting the directory only slows the next run.
+- An entry is written only after a later verification step passes
+  (`expect`, `locator.waitFor`, `agent.assert`, `agent.waitFor`). An `act`
+  nothing checks is never replayed.
+- `e2e init` gitignores `.e2e/cache/`; committing entries is opt-in. Remove
+  that line to share replays with CI and teammates (CI stays `read-only`
+  unless `cache: 'read-write'` is set explicitly).
+- A failing run evicts the entries it implicates. To rule the cache out of a
+  failure, run with `--no-cache`.
+
+## Inspect what the model did
+
+```bash
+npx --no-install e2e run tests/checkout.e2e.ts --debug      # step table, transcripts as artifacts
+npx --no-install e2e run tests/checkout.e2e.ts --ai-trace   # writes .e2e/ai-trace.json
+npx unbox-ai runs .e2e/ai-trace.json                        # one line per agent step
+npx unbox-ai summary .e2e/ai-trace.json --run 0             # turns, tokens, tool calls of one step
+```
+
+Never read `.e2e/ai-trace.json` directly; it is megabytes of resent context.
+Use `--no-cache` when the whole flow should be traced, since replayed steps
+make no model calls.
+
+## Beyond the built-in agent
+
+- `createAgent({ tools: { seedCart } })` adds AI SDK tools wrapped with
+  `defineTool(tool({ ... }), { mutates: true })` from `@e2edev/e2e/agent`, so
+  a flow can call a test API mid-step.
+- `createToolLoopExecutor` keeps the loop and replaces the prompt and the
+  tool vocabulary.
+- Any object implementing `StepExecutor` (`{ name, version, cache, runStep(ctx) }`)
+  can be the `agent`; the runner still owns observations, actions, budgets,
+  and the report.
+
+Full reference: https://e2e.docs.buildwithfern.com/agents
+
+## In CI
+
+Deterministic tests gate merges; agentic tests are opt-in. Keep them in a
+separate config (`e2e.agent.config.ts` with its own `tests` glob and a larger
+`timeout`), run them on a schedule or `workflow_dispatch`, and pass the model
+through `env: { E2E_MODEL, E2E_MODEL_API_KEY }` from CI variables and
+secrets.

package/skills/e2e/references/debugging.md

@@ -0,0 +1,81 @@
+# Debugging a failing run
+
+## Read the failure
+
+1. The `list` reporter ends with a `Failed Tests` section: each failure shows
+   the error code, the message, the test's failing line, and a code frame.
+   The summary's `Report` line names `.e2e/report.json`.
+2. `.e2e/report.json` has everything:
+
+```bash
+jq '.run | {status, exitCode, errors}' .e2e/report.json                  # run-level failures
+jq '.run.results[] | select(.status != "passed") | {titlePath, file, status}' .e2e/report.json
+jq '.run.results[] | select(.status != "passed") | .attempts[-1]
+    | {status, error, steps: [.steps[] | select(.status != "passed") | {api, label, status, error}], artifacts}' .e2e/report.json
+```
+
+3. Artifacts named there live under `.e2e/artifacts/`: screenshots, a
+   Playwright `trace.zip` per attempt (`npx playwright show-trace <file>`),
+   downloads, and with `--debug` the transcript of every agent step.
+
+## Error codes and what to do
+
+| Code | Usual cause | Fix |
+| --- | --- | --- |
+| `CONFIG_LOAD_FAILED` mentioning ES modules | `package.json` lacks `"type": "module"` | `npm pkg set type=module`, or rename the config to `.mts` and keep the tests in an ESM package |
+| `INVALID_CONFIG` | Unknown key or a stale shape: a top-level `app`, `defineConfig`, a `backend` key, `json` combined with `list` reporters | Move app options into `playwright({ ... })`; use `satisfies E2EConfig`; the message names the key |
+| `CONFIG_NOT_FOUND`, `CONFIG_AMBIGUOUS` | Wrong `--config` path; both `.ts` and `.mts` present | Fix the path; keep one config file |
+| `NO_TESTS` | The glob or a positional matched nothing | The message names each positional that matched nothing. Check `tests` in the config and the `.e2e.ts` suffix |
+| `COLLECTION_ERROR` | `async` describe body, `test.setup` inside `describe`, an option forbidden in a serial group, registration outside collection | Rework the structure per `writing-tests` |
+| `APP_UNREACHABLE` | `command` never answered `readyUrl` within `startupTimeout`; a service exited non-zero | Set `command.log` and read it; check the port and `url`; pass the env the app needs through `command.env`; raise `startupTimeout` |
+| `APP_ALREADY_RUNNING` | Something already serves `url` when the runner wanted to start `command` | Stop it, or set `reuseExisting: true` for local runs |
+| `APP_URL_REQUIRED` | A navigation on an engine without `url` | Add `url` to `playwright({ ... })` |
+| `LOCATOR_NOT_FOUND` | Wrong role or name, text not exact, element off screen or inside an iframe, page not open | Read the markup for the accessible name; try `exact: false` or a RegExp; `web.frameLocator` for iframes; `app.open()` first; `--headed` to look |
+| `LOCATOR_AMBIGUOUS` | Two matches: a hidden duplicate, a repeated label | Add `{ name }`, scope under a container, `filter`, `first()`, or `{ visible: true }` |
+| `ASSERTION_FAILED` | The expectation is wrong, or the state settles later than 5 s; for `agent.assert`, the judgment was false (explanation in the report) | Compare with the actual text in the report or screenshot; `{ timeout }` on the matcher; rewrite the question |
+| `ACTION_FAILED` | Element not actionable (covered, disabled, detached) or an operation timed out | Wait on the right condition with `expect` first; close overlays; check `actionTimeout` |
+| `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` for slow flows and agent steps |
+| `MODEL_UNAVAILABLE` | No `E2E_MODEL`, no key, or an unknown provider | Export `E2E_MODEL=provider/model-id` and `E2E_MODEL_API_KEY`; confirm the model id |
+| `MODEL_PROVIDER_FAILED` | Network, 5xx, rate limit, or no credits after the transport retries | Check the key and the quota; retry; exit code 3 |
+| `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | The goal was too big or ambiguous, or the provider slow | Split the goal, use on-screen wording, add `context`, raise `timeout` and `actionTimeout`, `--debug` to read the transcript |
+| `POLICY_DENIED` | Navigation outside `allowedOrigins`; a `Secret` given to a sink that is not a password field; reading a secure field | Add the origin to `allowedOrigins`; fill secrets only into password inputs; assert the outcome instead of the value |
+| `UNSUPPORTED_CAPABILITY` | A fixture the engine does not contribute (`web` on a device), `schema` or `vision` on `act`, an action the surface lacks | Declare `requires: ['web']`; drop the option; use a supported action |
+| `SESSION_UNAVAILABLE`, `SESSION_CONTRACT` | `session: 'x'` with no setup saving `x`; a setup that did not save every declared name | Add or fix the `test.setup` |
+| `ONLY_IN_CI` | `test.only` reached CI | Remove it |
+| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download or launch failed | `npx playwright install chromium --with-deps`; raise `launchTimeout` on slow machines |
+| `AUTH_CREDENTIAL_UNAVAILABLE` | `credentials.user('x')` for an undeclared name | Add it to `config.credentials` |
+
+## Tools
+
+| Do | When |
+| --- | --- |
+| `--headed` | Watch the flow; confirm what the page shows at the failing step |
+| `--workers 1 --retries 0` | Remove parallelism and retries from the picture |
+| `--no-cache` | Rule out a stale `agent.act` replay |
+| `--debug` | Read each agent step's duration, model calls, cost, and transcript |
+| `--ai-trace`, then `npx unbox-ai runs .e2e/ai-trace.json` | See exactly what the model was shown and called |
+| `command.log: '.e2e/logs/app.log'` | Read the app's own output when it never becomes ready or errors mid-test |
+| `await app.screenshot('before-submit')` | Attach evidence at a chosen point |
+| `CI=1 npx --no-install e2e run` | Reproduce CI-only behaviour: `ONLY_IN_CI`, read-only cache, `reuseExisting` ignored |
+
+## Flaky tests
+
+- A read (`textContent()`, `count()`) captured a value mid-update: replace
+  it with a matcher.
+- Shared data between tests: unique names per run, cleanup in `afterEach`,
+  or a `serial` group.
+- The app is not ready: assert on the element you are about to use rather
+  than on the previous page.
+- An agent judgment asserts an exact phrasing: judge the fact, and add a
+  deterministic `expect` beside it.
+- Timing under load: `retries` masks the cause; `--workers 1` and
+  `--headed` usually show it.
+
+## Is it the app?
+
+A deterministic step that fails every run at the same place with the same
+code is a product bug or a changed screen, not flakiness. Reproduce once
+with `--headed`, then fix the app, or update the locator and the expectation
+together. A blocked agent step (`AUTH_CREDENTIAL_UNAVAILABLE`,
+`ENVIRONMENT_UNAVAILABLE`, `SEED_DATA_MISSING`) exits 2 or 3 on purpose: fix
+the environment, not the test.

package/skills/e2e/references/running.md

@@ -0,0 +1,153 @@
+# Running tests
+
+## Commands
+
+```bash
+npx --no-install e2e run [files...] [options]   # run tests
+npx --no-install e2e list [files...] [options]  # print what run would select, without running
+npx --no-install e2e init [--yes]               # scaffold a project, refresh the agent skill
+npx --no-install e2e guide [topic]              # print this skill: setup, writing-tests, agent, running, debugging
+npx --no-install e2e cache ls|clear|stats       # read or empty the trace cache
+```
+
+`run` flags:
+
+| Flag | Effect |
+| --- | --- |
+| `[files...]` | Files, directories, or quoted globs relative to the project root. They narrow the config `tests` glob, never bypass it. |
+| `--config <path>` | Explicit config file. Default: `e2e.config.ts` or `.mts` found upward from the working directory. |
+| `--target <ids>` | Comma-separated target names. |
+| `--tag <tag>` | Repeatable tag filter; `--tag-mode all` requires every tag. |
+| `--headed` | Visible browser or simulator when the engine supports it. |
+| `--workers <n>`, `--retries <n>` | Override the resolved values. |
+| `--reporter <ids>` | `list`, `json`, `junit`, comma-separated. `json` cannot combine with `list`. |
+| `--artifacts <dir>` | Artifact root, default `.e2e/artifacts`. |
+| `--no-cache` | Run with the trace cache off. |
+| `--pass-with-no-tests` | Exit 0 when nothing matches instead of `NO_TESTS`. |
+| `--debug` | Phase timings and an agent step table on stderr; step transcripts saved as artifacts. |
+| `--ai-trace` | Record every model call to `.e2e/ai-trace.json`. |
+
+```bash
+npx --no-install e2e run tests/signup.e2e.ts
+npx --no-install e2e run tests/agent --tag smoke
+npx --no-install e2e run 'tests/**/*.smoke.e2e.ts' --target chromium --workers 1 --retries 0
+CI=1 npx --no-install e2e run   # reproduce the CI defaults locally
+```
+
+`list` takes the same files and the selection flags (`--config`, `--target`,
+`--tag`, `--tag-mode`, `--pass-with-no-tests`) and prints one line per
+test-target pair, `file › title [target]`, then exits without starting the
+app, an engine, or a worker. `--reporter json` prints `{ "pairs": [...] }`.
+Use it to check a filter before a run.
+
+```bash
+npx --no-install e2e list --tag smoke
+npx --no-install e2e list tests/signup.e2e.ts --reporter json
+```
+
+A `package.json` script keeps it short: `"test:e2e": "e2e run"`, then
+`pnpm test:e2e tests/signup.e2e.ts`.
+
+## The trace cache
+
+Entries live under `.e2e/cache/`, one file per key, named after the key
+digest. `cache` commands read the same config as `run`, so `--config` and
+`cache.dir` point them at the right store.
+
+| Command | Prints |
+| --- | --- |
+| `e2e cache ls` | One row per entry: test, target, instruction digest, age, action count. |
+| `e2e cache stats` | Directory, entry count, total size. |
+| `e2e cache clear` | Deletes the entries and the directory; files the runner never wrote stay. |
+
+Use `ls` to see what a committed cache would replay, and `clear` when a
+recorded flow is stale — `--no-cache` only skips the cache for one run.
+
+## Output
+
+- `list` (default): one line per file and target, a `Failed Tests` section
+  with each error, its code, the failing line and a code frame, then a
+  summary (`Test Files`, `Tests`, `AI`, `Duration`, `Report`).
+- `.e2e/report.json` is written on every run whatever the reporters:
+  `run.status`, `run.exitCode`, `run.errors[]` (run-level failures such as
+  `APP_UNREACHABLE`), and `run.results[]`, one per test and target, with
+  `titlePath`, `file`, `source`, `status`, and `attempts[]` holding `steps[]`,
+  `artifacts[]`, and `error`.
+- `junit`: `.e2e/junit.xml` beside the report, for CI test summaries.
+  Combine it with the terminal output: `--reporter list,junit`.
+- `json`: the report document on stdout.
+- Artifacts (screenshots, Playwright traces, `--debug` transcripts,
+  downloads) live under `.e2e/artifacts/`; every path is recorded in the
+  report.
+
+## Exit codes
+
+| Code | Meaning |
+| ---: | --- |
+| 0 | Every selected test passed, was flaky, or was skipped |
+| 1 | A test or setup test failed or timed out |
+| 2 | CLI, config, collection, credential, model-config, or policy error |
+| 3 | Engine, app process, model provider, artifact, or cleanup failure |
+| 4 | Internal runner error |
+| 130 | Interrupted |
+
+The highest code present wins for a mixed run (`130 > 4 > 3 > 2 > 1 > 0`).
+Do not retry a job on exit 2: it is deterministic. Exit 3 is the only one
+where a job-level retry makes sense.
+
+Ctrl-C once interrupts and still writes the report; twice forces teardown;
+three times exits at once.
+
+## Continuous integration
+
+CI mode is on when `CI` is set (and not `0` or `false`). It changes
+`retries` to 1, `workers` to 1, rejects `test.only` with `ONLY_IN_CI`, makes
+the trace cache `read-only`, and ignores `reuseExisting`.
+
+```yaml
+# .github/workflows/e2e.yml
+name: e2e
+on:
+  pull_request:
+  push:
+    branches: [main]
+permissions:
+  contents: read
+jobs:
+  e2e:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 26
+          cache: pnpm
+      - run: pnpm install --frozen-lockfile
+      - run: npx playwright install chromium --with-deps
+      - run: npx --no-install e2e run --reporter list,junit
+        env:
+          E2E_USER_ADMIN_USERNAME: ${{ secrets.E2E_USER_ADMIN_USERNAME }}
+          E2E_USER_ADMIN_PASSWORD: ${{ secrets.E2E_USER_ADMIN_PASSWORD }}
+      - if: ${{ !cancelled() }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-report
+          path: |
+            .e2e/report.json
+            .e2e/junit.xml
+      - if: failure()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-artifacts
+          path: .e2e/artifacts
+          retention-days: 7
+```
+
+- Install browsers as their own step so the download never counts against a
+  launch timeout.
+- Start the app through the engine's `command`; the runner tears it down on
+  every exit path.
+- Agentic suites: a separate config, run on `schedule` or
+  `workflow_dispatch`, `E2E_MODEL` as a CI variable and `E2E_MODEL_API_KEY`
+  as a secret, never a required check.