e2e 0.15.1 → 0.15.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (87) hide show
  1. package/README.md +51 -24
  2. package/dist/.build.json +1 -1
  3. package/dist/agent/act.js +3 -3
  4. package/dist/agent/act.js.map +1 -1
  5. package/dist/agent/error.d.ts +16 -3
  6. package/dist/agent/error.d.ts.map +1 -1
  7. package/dist/agent/error.js +25 -6
  8. package/dist/agent/error.js.map +1 -1
  9. package/dist/agent/model/tool-choice.d.ts +24 -1
  10. package/dist/agent/model/tool-choice.d.ts.map +1 -1
  11. package/dist/agent/model/tool-choice.js +27 -1
  12. package/dist/agent/model/tool-choice.js.map +1 -1
  13. package/dist/agent/step-cache.d.ts +14 -3
  14. package/dist/agent/step-cache.d.ts.map +1 -1
  15. package/dist/agent/step-cache.js +13 -3
  16. package/dist/agent/step-cache.js.map +1 -1
  17. package/dist/agent/tool-loop.d.ts.map +1 -1
  18. package/dist/agent/tool-loop.js +62 -9
  19. package/dist/agent/tool-loop.js.map +1 -1
  20. package/dist/cache/context.d.ts +19 -6
  21. package/dist/cache/context.d.ts.map +1 -1
  22. package/dist/cache/context.js +15 -21
  23. package/dist/cache/context.js.map +1 -1
  24. package/dist/cli/init/scaffold.d.ts +1 -0
  25. package/dist/cli/init/scaffold.d.ts.map +1 -1
  26. package/dist/cli/init/scaffold.js +2 -1
  27. package/dist/cli/init/scaffold.js.map +1 -1
  28. package/dist/cli/init/sibling-versions.json +1 -1
  29. package/dist/expect/values.js +1 -1
  30. package/dist/expect/values.js.map +1 -1
  31. package/dist/run/execute.d.ts.map +1 -1
  32. package/dist/run/execute.js +21 -8
  33. package/dist/run/execute.js.map +1 -1
  34. package/dist/types.d.ts +1 -1
  35. package/dist/types.d.ts.map +1 -1
  36. package/docs/agent-steps.mdx +126 -0
  37. package/docs/agents.mdx +217 -0
  38. package/docs/api-testing.mdx +65 -0
  39. package/docs/authentication.mdx +345 -0
  40. package/docs/browser.mdx +440 -0
  41. package/docs/bug-bash.mdx +90 -0
  42. package/docs/cache.mdx +306 -0
  43. package/docs/ci.mdx +210 -0
  44. package/docs/coding-agents.mdx +183 -0
  45. package/docs/debugging.mdx +291 -0
  46. package/docs/executors.mdx +243 -0
  47. package/docs/explore.mdx +174 -0
  48. package/docs/github.mdx +184 -0
  49. package/docs/index.mdx +82 -0
  50. package/docs/integrations/eas.mdx +216 -0
  51. package/docs/integrations/index.mdx +24 -0
  52. package/docs/integrations/kernel.mdx +163 -0
  53. package/docs/migrate/cypress.mdx +124 -0
  54. package/docs/migrate/detox.mdx +140 -0
  55. package/docs/migrate/maestro.mdx +229 -0
  56. package/docs/migrate/playwright.mdx +427 -0
  57. package/docs/migrate/selenium.mdx +110 -0
  58. package/docs/mobile-ci.mdx +567 -0
  59. package/docs/mobile.mdx +555 -0
  60. package/docs/models.mdx +241 -0
  61. package/docs/quickstart.mdx +260 -0
  62. package/docs/reference/agent.mdx +524 -0
  63. package/docs/reference/app.mdx +72 -0
  64. package/docs/reference/cli.mdx +817 -0
  65. package/docs/reference/config.mdx +986 -0
  66. package/docs/reference/engine.mdx +1362 -0
  67. package/docs/reference/environment.mdx +113 -0
  68. package/docs/reference/errors.mdx +143 -0
  69. package/docs/reference/expect.mdx +513 -0
  70. package/docs/reference/mcp.mdx +202 -0
  71. package/docs/reference/mobile.mdx +522 -0
  72. package/docs/reference/reporters.mdx +216 -0
  73. package/docs/reference/screen.mdx +688 -0
  74. package/docs/reference/test.mdx +449 -0
  75. package/docs/reference/web.mdx +611 -0
  76. package/docs/security.mdx +292 -0
  77. package/docs/starting-your-app.mdx +231 -0
  78. package/docs/subscriptions.mdx +157 -0
  79. package/docs/telemetry.mdx +185 -0
  80. package/docs/tools.mdx +150 -0
  81. package/docs/viewports.mdx +146 -0
  82. package/docs/writing-an-engine.mdx +316 -0
  83. package/docs/writing-tests.mdx +256 -0
  84. package/package.json +2 -1
  85. package/skills/e2e/SKILL.md +4 -1
  86. package/skills/e2e/references/agent.md +3 -1
  87. package/skills/e2e/references/setup.md +13 -4
@@ -0,0 +1,174 @@
1
+ ---
2
+ title: Exploring without a test
3
+ description: Give the agent a goal instead of a test file. It plans its own steps, drives the app, and reports what it finds.
4
+ ---
5
+
6
+ Use `e2e explore` to investigate a flow without writing a test. Give the
7
+ agent a goal; it chooses the steps and records findings in
8
+ `.e2e/report.json`.
9
+
10
+ <CodeGroup>
11
+ ```bash npm
12
+ npx e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
13
+ ```
14
+
15
+ ```bash pnpm
16
+ pnpm exec e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
17
+ ```
18
+
19
+ ```bash bun
20
+ bunx e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
21
+ ```
22
+ </CodeGroup>
23
+
24
+ Explore can help you investigate a feature or choose flows to turn into
25
+ regression tests. It uses the same model, tools, and action rules as
26
+ `agent.act()`. Review the findings against their evidence.
27
+
28
+ ## What a run does
29
+
30
+ The runner opens the target app, plans one step, and carries it out with
31
+ `agent.act`. The agent records findings as it goes, including what it
32
+ expected, what it saw, and how to reproduce it. A screenshot is attached
33
+ when capture is allowed and succeeds.
34
+
35
+ The process repeats until the planner considers the goal covered or a
36
+ budget expires, then produces a closing assessment. A failed step does not
37
+ stop exploration. Three consecutive failed or blocked steps with no new
38
+ findings end the run.
39
+
40
+ ## Budgets
41
+
42
+ | Budget | Default | Range | Flag |
43
+ | --- | --- | --- | --- |
44
+ | Exploration steps | 8 | 1 to 12 | `--max-steps <n>` |
45
+ | Wall clock | 10 minutes | 3 to 15 minutes | `--timeout <ms>` |
46
+ | Actions and model calls per step | 40 each, unless the agent sets `maxSteps` or `maxModelCalls` | 1 to 100 | config |
47
+
48
+ Start with the defaults. Increase the step or time budget when the report
49
+ shows that the run stopped before reaching the part of the flow you need.
50
+
51
+ ## The verdict
52
+
53
+ | Outcome | Run status | Exit code |
54
+ | --- | --- | --- |
55
+ | Steps ran, no `issue` finding, and not every step was blocked | passed | 0 |
56
+ | At least one `issue` finding | failed | 1 |
57
+ | No step ran and nothing was found | blocked | 1 |
58
+ | Every step was blocked, with no `issue` finding | blocked | 1, 2, or 3, from the first blocker |
59
+ | Config, engine, or model provider error | error | 2 or 3 |
60
+
61
+ An `issue` is a functional defect and fails the run. A `warning` is cosmetic
62
+ and does not. If every step was blocked, the run stays blocked and reports
63
+ the first blocker's error. A step that reaches its action or time budget
64
+ is recorded as time-boxed; findings still determine the verdict.
65
+
66
+ ## What you get
67
+
68
+ The terminal follows the exploration as it happens: each finding the moment
69
+ the agent reports it, and each step as it finishes, by title.
70
+
71
+ ```text
72
+ ❯ web Exploring Explore this bookshop like a careful first-time buyer: browse the catalog, ...
73
+ ⚑ medium issue Home greeting exposes an unresolved userName template (/)
74
+ ⚑ high issue Neuromancer shows an impossible negative stock availability (/catalog)
75
+ ✓ 1 Catalog browsing 31.46s · 7 actions · 4 findings
76
+ ⚑ high issue Cart total does not update when Hyperion quantity increases (/cart)
77
+ ⚑ trivial warning Checkout newsletter label misspells Receive (/checkout)
78
+ ✓ 2 Cart quantities and removal 19.47s · 4 actions · 3 findings
79
+ ended: the agent covered the goal
80
+ ```
81
+
82
+ The final report lists issues before warnings, with the most severe first.
83
+ Each finding includes its location, expected and observed behavior,
84
+ reproduction steps, and any screenshot evidence. Severity runs from
85
+ `trivial` at 1 to `critical` at 5.
86
+
87
+ The JSON report stores these details under `run.explore`, along with the
88
+ goal, budgets, steps, assessment, and reason the run ended. Findings carry no
89
+ screenshots once a secret is filled; see
90
+ [screenshots after a secret fill](/security#screenshots-after-a-secret-fill).
91
+
92
+ Its artifacts land in a directory named after the goal, such as
93
+ `.e2e/artifacts/web/explore-check-the-cart-totals-1a2b3c4d5e6f7a8b/`. A rerun
94
+ of the same goal writes to the same directory.
95
+
96
+ `--reporter list,markdown` also writes the findings to `.e2e/summary.md`.
97
+ `--debug`, `--ai-trace`, `--trace`, and `--video` work as they do for
98
+ `e2e run`: they apply when the engine can record, with a notice otherwise.
99
+ The exploration is one attempt, so a retry mode (`on-first-retry`,
100
+ `on-all-retries`, and the CI trace default) records nothing, and the run
101
+ prints a notice saying to pass `--trace on` (or `--video on`); put the goal
102
+ before a bare `--trace` or `--video`.
103
+
104
+ ## Your config applies
105
+
106
+ Explore reads `e2e.config.ts`, including the target, app command, model, and
107
+ agent context. It starts the target's app command. Use `--agent <name>` to select a configured agent.
108
+ The agent's `tools` and `system` instructions also apply. Custom executors
109
+ are replaced by the built-in agent for exploration, with a notice naming the
110
+ agent. The built-in agent needs a model: an entry with none, a custom
111
+ executor that brought none included, fails with `MODEL_UNAVAILABLE` before
112
+ anything starts.
113
+
114
+ Every configured credential is available to the explorer. It receives
115
+ account names and usernames; passwords remain secret handles and require
116
+ an engine that supports sensitive fills. Configure only the accounts needed
117
+ for the exploration. Without credentials, it skips password sign-in.
118
+
119
+ Use `--target <name>` to choose among several targets. Otherwise, explore
120
+ uses the first and prints a notice. Exploration disables the replay cache
121
+ and retries.
122
+
123
+ ## Start signed in
124
+
125
+ Pass `--session <name>` to explore as a signed-in user. The name is a session
126
+ a [setup test](/authentication) in your test files saves:
127
+
128
+ <CodeGroup>
129
+ ```bash npm
130
+ npx e2e explore --session admin 'Explore the admin settings and find bugs'
131
+ ```
132
+
133
+ ```bash pnpm
134
+ pnpm exec e2e explore --session admin 'Explore the admin settings and find bugs'
135
+ ```
136
+
137
+ ```bash bun
138
+ bunx e2e explore --session admin 'Explore the admin settings and find bugs'
139
+ ```
140
+ </CodeGroup>
141
+
142
+ Explore runs the one setup test that declares the session, restores the
143
+ session, and opens the app when the target declares a URL, as for a test
144
+ that declares `{ session: 'admin' }`. No other test runs. The setup test
145
+ runs as it does for `e2e run`, with your configured agents, replay cache,
146
+ and retries. The explorer is told the app starts signed in, so it does not
147
+ plan a sign-in step.
148
+
149
+ If no setup test declares the name, the run stops with `COLLECTION_ERROR`
150
+ before any app process starts. The message lists the sessions your setup
151
+ tests declare.
152
+
153
+ A setup that fills a password leaves the exploration without screenshots. To
154
+ keep them, sign in without filling a secret; see
155
+ [screenshots after a secret fill](/security#screenshots-after-a-secret-fill).
156
+
157
+ ## Writing a good goal
158
+
159
+ Name the flow and the user's goal, for example, "Explore checkout as a
160
+ first-time buyer and check that totals match the cart." Include boundaries
161
+ when needed, such as "stop before placing the order." Without a goal,
162
+ explore uses `Explore the app and find bugs`.
163
+
164
+ To cover a whole app or branch, run several explorations at once and verify
165
+ what they find: see [Bug bashes](/bug-bash).
166
+
167
+ <CardGroup cols={2}>
168
+ <Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-explore">
169
+ Every flag of `e2e explore`.
170
+ </Card>
171
+ <Card title="Agents and personas" icon="users" href="/agents">
172
+ The agents explore can run as.
173
+ </Card>
174
+ </CardGroup>
@@ -0,0 +1,184 @@
1
+ ---
2
+ title: Pull request comments
3
+ description: The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.
4
+ ---
5
+
6
+ `@e2e-dev/github` posts test results as a pull request comment, including
7
+ failures, source links, and links to workflow artifacts. Reruns update the
8
+ same comment, and a `--last-failed` rerun updates it as one run. The reporter
9
+ also writes the results to the job summary.
10
+
11
+ ## Setup
12
+
13
+ <CodeGroup>
14
+ ```bash npm
15
+ npm install --save-dev @e2e-dev/github
16
+ ```
17
+
18
+ ```bash pnpm
19
+ pnpm add -D @e2e-dev/github
20
+ ```
21
+
22
+ ```bash bun
23
+ bun add -d @e2e-dev/github
24
+ ```
25
+ </CodeGroup>
26
+
27
+ ```ts title="e2e.config.ts"
28
+ import type { E2EConfig } from 'e2e';
29
+ import { web } from '@e2e-dev/web';
30
+ import { github } from '@e2e-dev/github';
31
+
32
+ export default {
33
+ targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
34
+ reporters: ['list', github()],
35
+ } satisfies E2EConfig;
36
+ ```
37
+
38
+ Give the workflow permission to write pull request comments and pass
39
+ `GITHUB_TOKEN` to the test step:
40
+
41
+ ```yaml title=".github/workflows/e2e.yml"
42
+ permissions:
43
+ contents: read
44
+ pull-requests: write
45
+
46
+ jobs:
47
+ e2e:
48
+ steps:
49
+ # ...checkout, install, browsers
50
+ - name: Run e2e
51
+ run: npx e2e run --reporter list,junit
52
+ env:
53
+ GITHUB_TOKEN: ${{ github.token }}
54
+ - name: Upload artifacts
55
+ if: ${{ !cancelled() }}
56
+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
57
+ with:
58
+ name: e2e-artifacts
59
+ path: .e2e/artifacts
60
+ ```
61
+
62
+ Upload artifacts even when a retry passes. Each failure's evidence links open
63
+ the run's Artifacts section; inside the download, every test has a directory
64
+ named after it and the agent that ran it
65
+ (`web/tests_billing.e2e.ts__upgrades/default/attempt-0/`), so keep
66
+ `path: .e2e/artifacts` and the download lines up with the report. A project
67
+ below the checkout root, such as `packages/e2e-tests`, gets source links under
68
+ that path. The checkout root is the nearest directory holding `.git` inside
69
+ `GITHUB_WORKSPACE`, so `actions/checkout` with a `path` input works too.
70
+
71
+ ## The comment
72
+
73
+ ```markdown
74
+ ### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
75
+ 9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31
76
+
77
+ **🔴 billing › upgrades to Pro**
78
+ [tests/billing.e2e.ts:12](https://github.com/acme/shop/blob/1a2b3c/tests/billing.e2e.ts#L12)
79
+
80
+ **ASSERTION_FAILED** at step 4 of 6: `agent.act` "Pick the Pro plan and pay", after 38.0s and 12 model calls
81
+
82
+ > expected heading "Your cart" to be visible
83
+ > The Pay button opened a page that still shows the Free plan.
84
+
85
+ - Failed the same way on both attempts: **ASSERTION_FAILED** at step 4.
86
+ - Turn 9: `tap({"target":"n41"})` → Screen changed: /checkout/plan. New: #n50 heading "Free plan"
87
+ - Screen: `/checkout/plan`
88
+
89
+ Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [video](https://github.com/acme/shop/actions/runs/123#artifacts), [trace](https://github.com/acme/shop/actions/runs/123#artifacts)
90
+
91
+ <details>
92
+ <summary>⚠️ 1 flaky test passed on a retry</summary>
93
+
94
+ **⚠️ invite a member by email**
95
+ [tests/team.e2e.ts:21](https://github.com/acme/shop/blob/1a2b3c/tests/team.e2e.ts#L21)
96
+
97
+ **STEP_TIMEOUT** at step 3 of 3: `agent.waitFor` "the invitation email", after 30.0s
98
+
99
+ > the invitation email took too long
100
+
101
+ Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [trace](https://github.com/acme/shop/actions/runs/123#artifacts)
102
+ </details>
103
+
104
+ <details>
105
+ <summary>All 6 tests in 5 files</summary>
106
+
107
+ | | Test | Agent | Time |
108
+ | --- | --- | --- | --- |
109
+ | 🔴 | **tests/billing.e2e.ts** · 1 failed | 3 steps · 19 calls | 41.2s |
110
+ | 🔴 | billing › upgrades to Pro | 3 steps · 19 calls | 41.2s |
111
+ | ⚠️ | **tests/team.e2e.ts** · 1 flaky | 1 step | 6.4s |
112
+ | ⚠️ | invite a member by email (1 failed attempt first) | 1 step | 6.4s |
113
+ ...
114
+ </details>
115
+
116
+ <sub>e2e 0.13.0 · 2m 15s · web · [run artifacts](https://github.com/acme/shop/actions/runs/123#artifacts)</sub>
117
+ ```
118
+
119
+ The comment shows test counts and agent usage, followed by each failure's
120
+ error, source line, and evidence links. Flaky tests and the full test list
121
+ are collapsed; the list is one table with a row per file above its tests,
122
+ and its agent column appears only when the run used the agent. Run-level
123
+ errors appear above the test failures. Large
124
+ reports are shortened to fit GitHub's comment limit. Text from the app under
125
+ test that GitHub would turn into a mention, an issue link, or a URL is shown
126
+ as code, so a screen reading `@someone` or `https://...` notifies no one and
127
+ links nowhere.
128
+
129
+ The built-in `markdown` reporter writes the same summary to
130
+ `.e2e/summary.md`.
131
+
132
+ ## Rerunning the failed tests
133
+
134
+ A step that gives failed tests a second chance keeps one comment for both
135
+ passes:
136
+
137
+ ```yaml
138
+ - name: Run e2e
139
+ run: npx e2e run || npx e2e run --last-failed
140
+ env:
141
+ GITHUB_TOKEN: ${{ github.token }}
142
+ ```
143
+
144
+ The comment covers the whole suite, not only the tests the rerun ran again.
145
+ A test that failed and then passed shows as flaky, a test that failed twice
146
+ stays failed, and the rest keep their first-pass results. The comment
147
+ matches the job's status: it stays red when another filter kept a failed
148
+ test out of the rerun.
149
+
150
+ ## Matrix jobs
151
+
152
+ The reporter identifies a comment by project, workflow, and job. Matrix
153
+ jobs share those identifiers, so give each matrix entry a distinct `key`:
154
+
155
+ ```ts
156
+ reporters: ['list', github({ key: process.env.MATRIX_BROWSER })],
157
+ ```
158
+
159
+ Set `MATRIX_BROWSER: ${{ matrix.browser }}` in the step's `env`. The key also
160
+ appears in the headline, such as `e2e chromium: 77 passed`. Keep it stable
161
+ across runs so reruns update the existing comment.
162
+
163
+ ## When nothing is posted
164
+
165
+ The reporter never changes the run's status or exit code. When it cannot
166
+ post, it says why in one row under the summary:
167
+
168
+ ```
169
+ Report .e2e/report.json
170
+
171
+ GitHub not posted: set GITHUB_TOKEN in the step's env (GITHUB_TOKEN: ${{ github.token }}); written to the job summary
172
+ ```
173
+
174
+ | Situation | What happens |
175
+ | --- | --- |
176
+ | Not on GitHub Actions | Nothing is written; the row says so. |
177
+ | A `push`, `schedule`, or `workflow_dispatch` run | The job summary is written; there is no pull request to comment on. |
178
+ | No `GITHUB_TOKEN` in the step's `env` | The job summary is written; the row shows the line to add. `GH_TOKEN` is read too. |
179
+ | A `pull_request` event from a fork | Its token is read-only, so posting fails and the reporter explains why. |
180
+ | An `issue_comment` event | On a pull request's thread, the comment is posted without source links, since the checked-out commit is the default branch. |
181
+ | Missing `pull-requests: write` | The same failure, with the permission named. |
182
+
183
+ The token is read when the run finishes and is excluded from the report.
184
+ GitHub Enterprise Server uses `GITHUB_SERVER_URL` and `GITHUB_API_URL`.
package/docs/index.mdx ADDED
@@ -0,0 +1,82 @@
1
+ ---
2
+ title: Introduction
3
+ description: Open source AI testing for web and mobile apps. Spend less time on tests and more time building.
4
+ ---
5
+
6
+ import { VideoCard } from './snippets/video-card.jsx';
7
+
8
+ e2e is a next-generation testing framework for web and mobile
9
+ apps, built to keep up with the speed we ship software today.
10
+
11
+ You choose how agentic each test is: fully deterministic with
12
+ [locators](/writing-tests#deterministic-apis-screen-and-expect) and
13
+ [assertions](/reference/expect#locator-matchers), fully driven by an agent
14
+ from [goals](/writing-tests#goals-agent-act) in natural language, or a mix of
15
+ both in the same test.
16
+
17
+ <VideoCard
18
+ id="NEawZb6Hssw"
19
+ title="Watch e2e explained in 2 minutes"
20
+ subtitle="Learn how to chain agent goals with deterministic locator checks, run the same tests on web, iOS, and Android, and set up a project in a minute."
21
+ />
22
+
23
+ ```ts title="tests/checkout.e2e.ts"
24
+ import { test, expect } from 'e2e';
25
+
26
+ test('a member upgrades to Pro', async ({ app, agent, screen }) => {
27
+ await app.open('/settings/billing');
28
+
29
+ await agent.act('upgrade the workspace to the Pro plan');
30
+ await agent.assert('the invoice preview shows a prorated amount');
31
+
32
+ await expect(screen.getByRole('status')).toContainText('Pro');
33
+ });
34
+ ```
35
+
36
+ ## Why e2e
37
+
38
+ Testing should not slow down how fast you ship or how sure you are of what
39
+ you shipped.
40
+
41
+ - **Time back.** Write a goal instead of a script of selectors and waits.
42
+ When the UI changes, the agent adapts, so you spend less time writing and
43
+ fixing tests and more time building.
44
+ - **Less risk.** Cover more of the flows your users depend on, on web and
45
+ mobile, and catch regressions before they reach production.
46
+ - **Cheap runs.** Verified `agent.act` steps [replay from a cache](/cache)
47
+ with no model calls. Test more flows without the token bill or a slow CI.
48
+
49
+ ## How it works
50
+
51
+ - **Targets.** Each test runs once per target, such as Chromium or an iPhone
52
+ simulator. The [web engine](/starting-your-app) drives browsers through
53
+ Playwright, the [mobile engine](/mobile) drives iOS simulators and Android
54
+ emulators through agent-device, and you can
55
+ [write your own](/writing-an-engine).
56
+ - **Agent steps.** `agent.act` completes a goal. `agent.assert`,
57
+ `agent.waitFor`, and `agent.extract` judge the screen. Use your own
58
+ [subscription](/subscriptions), [API key, or local model](/models). Tests
59
+ without agent steps need no model.
60
+ - **Cache.** An `act` step that a later check verifies is recorded. The next
61
+ run replays it and hands back to the agent when the app changed. `assert`,
62
+ `waitFor`, and `extract` always call the model.
63
+ - **Reports.** The runner [redacts registered secrets](/security) from model
64
+ input and records steps, failures, and artifacts in a
65
+ [report](/debugging) you or your coding agent can read.
66
+
67
+ ## Get started
68
+
69
+ <CardGroup cols={2}>
70
+ <Card title="Quickstart" icon="rocket" href="/quickstart">
71
+ One prompt for your coding agent, or set up by hand.
72
+ </Card>
73
+ <Card title="Writing tests" icon="pen-to-square" href="/writing-tests">
74
+ Write goals, check outcomes, and extract data.
75
+ </Card>
76
+ <Card title="Coding agents" icon="robot" href="/coding-agents">
77
+ Let your coding agent write, run, and debug tests.
78
+ </Card>
79
+ <Card title="Mobile" icon="mobile-screen" href="/mobile">
80
+ Run tests on iOS simulators and Android emulators.
81
+ </Card>
82
+ </CardGroup>
@@ -0,0 +1,216 @@
1
+ ---
2
+ title: EAS Simulators
3
+ sidebarTitle: EAS Simulators
4
+ description: Run mobile targets on EAS Simulators' hosted iOS simulators and Android emulators with the easSimulators() device provider from @e2e-dev/eas.
5
+ ---
6
+
7
+ `easSimulators()` from `@e2e-dev/eas` is a
8
+ [device provider](/mobile#hosted-devices) for
9
+ [EAS Simulators](https://expo.dev/services/simulators). The run gets hosted
10
+ iOS simulators or Android emulators, started when it starts and stopped when
11
+ it ends, and the engine drives them through the agent-device daemon EAS runs
12
+ beside each one. Tests, config, and CI stay the same.
13
+
14
+ <Tip>
15
+ No Mac, Xcode, or Android SDK on the machine that runs e2e.
16
+ </Tip>
17
+
18
+ ## Setup
19
+
20
+ <CodeGroup>
21
+ ```bash npm
22
+ npm install --save-dev @e2e-dev/eas
23
+ ```
24
+
25
+ ```bash pnpm
26
+ pnpm add -D @e2e-dev/eas
27
+ ```
28
+
29
+ ```bash bun
30
+ bun add -d @e2e-dev/eas
31
+ ```
32
+ </CodeGroup>
33
+
34
+ EAS Simulators is in limited access; check yours with
35
+ `npx eas-cli@latest simulator:availability`.
36
+
37
+ ```ts title="e2e.config.ts"
38
+ import type { E2EConfig } from 'e2e';
39
+ import { mobile } from '@e2e-dev/mobile';
40
+ import { easSimulators } from '@e2e-dev/eas';
41
+
42
+ export default {
43
+ targets: [
44
+ {
45
+ name: 'ios',
46
+ engine: mobile({
47
+ platform: 'ios',
48
+ device: easSimulators({ buildId: process.env.EAS_BUILD_ID }),
49
+ videoTouches: false,
50
+ }),
51
+ app: { bundleId: 'com.example.app' },
52
+ },
53
+ ],
54
+ workers: 2,
55
+ } satisfies E2EConfig;
56
+ ```
57
+
58
+ The sessions belong to the Expo project the app config links:
59
+ `extra.eas.projectId`, the id `eas init` writes. The provider reads it from
60
+ the app config beside `e2e.config.ts`:
61
+
62
+ - **`app.json` or `app.config.json`**: read as JSON, from under `expo` when
63
+ the file has that key.
64
+ - **A dynamic config** (`app.config.ts`, `.js`, `.mjs`, `.cjs`, `.mts`, or
65
+ `.cts`): evaluated with the project's own `expo config --type public`,
66
+ in the run's environment, since the config can compute the id. As with
67
+ eas-cli, `.env` files are not loaded (`EXPO_NO_DOTENV=1`), so a variable
68
+ the config reads must be in the environment `e2e run` starts in. It needs
69
+ `expo` installed in the project.
70
+
71
+ It reads the config once per run, before the first session starts, and it
72
+ needs an `@e2e-dev/mobile` that passes `projectRoot` to device providers; an
73
+ older one fails the lease and asks for an upgrade or `projectId`. When
74
+ `e2e.config.ts` is not in the app's directory, or the app config links no
75
+ project, pass the id as `projectId` (the project's id, not its slug).
76
+
77
+ The package calls Expo's API with `fetch`, so there is no SDK to install.
78
+
79
+ ## Authenticate
80
+
81
+ The provider authenticates the way eas-cli does:
82
+
83
+ * **`EXPO_TOKEN`**, an [access token](https://expo.dev/settings/access-tokens)
84
+ of an account or robot with access to the project, in the environment
85
+ `e2e run` starts in. Use it in CI, as a secret.
86
+ * **The eas-cli login**, when `EXPO_TOKEN` is not set. On a machine where you
87
+ ran `eas login`, the provider reads the session eas-cli keeps in
88
+ `~/.expo/state.json` and needs no token. The home is the run's `HOME`, or
89
+ `USERPROFILE` on Windows.
90
+
91
+ `EXPO_TOKEN` wins when both are there. Its account is the one that starts the
92
+ sessions, so set it to run as a robot on a machine signed in as yourself.
93
+ Sessions always go to production `api.expo.dev`: eas-cli's staging and local
94
+ logins (`EXPO_STAGING`, `EXPO_LOCAL`) are not read.
95
+
96
+ On a CI runner or shared machine, set `EXPO_TOKEN`. Without it, a login someone
97
+ left in the runner's home starts billed sessions as their account.
98
+
99
+ ## Install the app
100
+
101
+ The simulator boots with no apps. Pick one way to put yours on it:
102
+
103
+ - **An EAS Build**: `buildId` names a simulator build (`"ios": { "simulator": true }`
104
+ in `eas.json`) or an APK, and EAS installs and launches it before the
105
+ session is ready. Leave the target's `app.appPath` out, and keep
106
+ `app.bundleId` so `app.open()` knows what to launch. The build never
107
+ leaves EAS.
108
+ - **A URL**: `applicationArchiveUrl` names an app archive EAS downloads and
109
+ installs the same way.
110
+ - **A local build**: the target's `app.appPath` and a fixture that calls
111
+ `device.installApp()`, as with a [local simulator](/mobile). The build
112
+ uploads through the session's daemon to each simulator. Leave `buildId`
113
+ and `applicationArchiveUrl` out.
114
+
115
+ ```ts
116
+ // EAS installs the build.
117
+ { engine: mobile({ platform: 'ios', device: easSimulators({ buildId }) }), app: { bundleId: 'com.example.app' } }
118
+
119
+ // A fixture uploads the build with device.installApp().
120
+ { engine: mobile({ platform: 'ios', device: easSimulators() }), app: { bundleId: 'com.example.app', appPath: 'build/App.app' } }
121
+ ```
122
+
123
+ ## Options
124
+
125
+ | Option | Default | Meaning |
126
+ | ----------------------- | ----------- | ------- |
127
+ | `projectId` | the app config's | The Expo project the sessions belong to. Absent, `extra.eas.projectId` of the app config. See [Setup](#setup). |
128
+ | `buildId` | none | EAS Build EAS installs and launches on every simulator. |
129
+ | `applicationArchiveUrl` | none | App archive URL EAS installs in place of a build. Excludes `buildId`. |
130
+ | `device` | EAS's pick | iOS device name or UDID (`'iPhone 17 Pro'`), or an Android AVD hardware profile (`'pixel_9'`). |
131
+ | `maxIdleTimeMinutes` | `10` | Minutes without an agent-device command after which EAS stops the session. EAS wants it below `maxDurationMinutes`, so a shorter duration lowers the default to one less. See [What the provider does](#what-the-provider-does). |
132
+ | `maxDurationMinutes` | account cap | Minutes a session may run once ready. EAS caps it at 40, or 115 on a high-priority plan. |
133
+ | `agentDeviceVersion` | the engine's | The agent-device version EAS starts the session's daemon at. Defaults to the one `@e2e-dev/mobile` pins, so the remote daemon answers as the engine's client expects; without it EAS starts the latest. |
134
+ | `tags` | `[]` | Tags on every session, beside the run's own. |
135
+
136
+ An option not in this table, `buildId` and `applicationArchiveUrl` together, or
137
+ a `maxIdleTimeMinutes` not below a `maxDurationMinutes` passed with it fails the
138
+ config load with `INVALID_CONFIG`.
139
+
140
+ ## Sessions
141
+
142
+ Each target starts up to `workers` sessions (`--workers` overrides it),
143
+ fewer when it has fewer test files, and tests start once all are ready.
144
+ A session takes about three minutes from request to ready (queue, boot,
145
+ build install), all in parallel, before the run's clock starts.
146
+
147
+ A run must fit in one session's lifetime: EAS stops a session
148
+ `maxDurationMinutes` after it is ready, 40 minutes by default on a
149
+ normal-priority plan and 115 on a high-priority one. Split a longer suite
150
+ across more workers or across CI jobs.
151
+
152
+ EAS runs as many sessions at once as the Expo account allows, and queues
153
+ the rest until one ends. None ends before the run does, so keep `workers`
154
+ times the number of EAS targets within that number. Every session queues
155
+ for a few seconds; one still queued after two minutes logs `still queued at
156
+ EAS after two minutes`. Targets prepare one after another, so the sessions of a target
157
+ that is ready wait idle while the next target's queue and boot. A session
158
+ still queued once another session of the run has been ready for
159
+ `maxIdleTimeMinutes` fails the lease, and the run stops every session: EAS
160
+ is stopping the idle one.
161
+
162
+ ## Watch a run
163
+
164
+ The provider logs each session's expo.dev page as it is created:
165
+
166
+ ```text
167
+ ℹ leasing 2 ios device(s) from eas-simulators
168
+ ℹ eas-simulators (1 of 2): simulator session 01a0…, https://expo.dev/accounts/acme/projects/shop/simulator-sessions/01a0…
169
+ ℹ eas-simulators: leased 01a0…
170
+ ```
171
+
172
+ A browser preview that opens without a token is logged too, as
173
+ `watch at <url>`. A preview that needs one is not: the token would give
174
+ anyone who reads the CI log control of the simulator.
175
+
176
+ Sessions are named `e2e <target> <n> of <total>` and tagged `e2e`,
177
+ `e2e-run:<run id>`, and `e2e-target:<target>`, so
178
+ `eas simulator:list --tag e2e-run:<run id>` finds a run's sessions.
179
+
180
+ ## Recordings
181
+
182
+ An attempt that records [video](/reference/config#video) records through
183
+ agent-device on the simulator, as it does locally, and saves
184
+ `video/video.mp4` under the attempt's artifact directory.
185
+
186
+ Set the mobile engine's `videoTouches: false`, as the setup above does.
187
+ agent-device draws its touch indicator into a recording when the recording
188
+ stops, and on an EAS iOS simulator that takes minutes, so every attempt that
189
+ records video runs out its `cleanupTimeout` stopping it and keeps no file.
190
+ Without the indicator, stopping takes about two seconds.
191
+
192
+ ## What the provider does
193
+
194
+ - Reads the project id from the app config once, before the first session,
195
+ when `projectId` is absent. A config that links no project fails the
196
+ lease before any session starts, naming the file.
197
+ - Starts each `agent-device` session and waits until its daemon is
198
+ reachable, then hands the engine the daemon's URL and token.
199
+ - Defaults `maxIdleTimeMinutes` to 10. A run that dies before the engine
200
+ stops its sessions leaves them idle, and EAS stops them after that long.
201
+ - Starts each daemon at the agent-device version `@e2e-dev/mobile` pins.
202
+ - Stops a session that errors, is cancelled, stays queued while another
203
+ session of the run idles to `maxIdleTimeMinutes`, or is still booting 15
204
+ minutes after leaving the queue, and treats one EAS no longer knows as
205
+ already stopped.
206
+ - Cancels the job of a session it stops while the job is still queued:
207
+ EAS would otherwise start it later and count it against the account's
208
+ concurrent sessions.
209
+ - Rides out up to three failed status polls in a row, a blip at
210
+ api.expo.dev, before giving a session up.
211
+
212
+ No `EXPO_TOKEN` and no eas-cli login fails the lease before any request to
213
+ EAS, with a message to run `eas login` or set `EXPO_TOKEN`. An account without
214
+ EAS Simulators, or a session that fails to start, fails the lease with EAS's
215
+ message. Either ends the run before any test, and the run's other sessions,
216
+ started or queued, are still stopped.
@@ -0,0 +1,24 @@
1
+ ---
2
+ title: Integrations
3
+ sidebarTitle: Overview
4
+ description: Official integrations with hosted services, one package per service.
5
+ ---
6
+
7
+ Official integrations connect e2e to hosted services through the engines'
8
+ provider seams, one package per service. Tests and CI stay the same, and the
9
+ config changes only where a target names its `browser` or `device`: the
10
+ integration decides where it comes from.
11
+
12
+ <CardGroup cols={2}>
13
+ <Card title="Kernel" icon="/images/integrations/kernel.svg" href="/integrations/kernel">
14
+ `@e2e-dev/kernel`: hosted Chromium for web targets, one browser per worker or per test.
15
+ </Card>
16
+ <Card title="EAS Simulators" icon="/images/integrations/expo.svg" href="/integrations/eas">
17
+ `@e2e-dev/eas`: hosted iOS simulators and Android emulators for mobile targets.
18
+ </Card>
19
+ </CardGroup>
20
+
21
+ A service without an integration is one small provider in your project:
22
+ see [Hosted browsers](/browser#hosted-browsers) for the web engine's
23
+ `BrowserProvider` and [Hosted devices](/mobile#hosted-devices) for the mobile engine's
24
+ `DeviceProvider`.