e2e 0.17.0 → 0.18.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (166) hide show
  1. package/README.md +10 -0
  2. package/dist/.build.json +1 -1
  3. package/dist/agent/ai-sdk.js +5 -4
  4. package/dist/agent/error.js +3 -3
  5. package/dist/agent/invocation.js +3 -4
  6. package/dist/agent/model/tool-choice.d.ts +9 -0
  7. package/dist/agent/model/tool-choice.js +14 -0
  8. package/dist/agent/observation-feed.js +3 -4
  9. package/dist/agent/phases.d.ts +7 -1
  10. package/dist/agent/phases.js +12 -3
  11. package/dist/agent/prompts.js +3 -0
  12. package/dist/agent/replay.js +18 -15
  13. package/dist/agent/step-cache.d.ts +2 -1
  14. package/dist/agent/step-cache.js +9 -5
  15. package/dist/agent/tool-loop.js +92 -5
  16. package/dist/cache/trace.js +6 -2
  17. package/dist/cli/bin.js +2 -2
  18. package/dist/cli/index.js +35 -24
  19. package/dist/cli/init/mcp-config.d.ts +2 -1
  20. package/dist/cli/init/mcp-config.js +8 -3
  21. package/dist/cli/init/sibling-versions.json +1 -1
  22. package/dist/cli/mcp.d.ts +3 -2
  23. package/dist/cli/mcp.js +52 -16
  24. package/dist/cli/run-output.d.ts +17 -0
  25. package/dist/cli/run-output.js +45 -0
  26. package/dist/collect/collect.js +8 -4
  27. package/dist/collect/registry.js +27 -16
  28. package/dist/config/compiled-files.d.ts +36 -0
  29. package/dist/config/compiled-files.js +50 -0
  30. package/dist/config/diagnose.js +21 -19
  31. package/dist/config/esm-hooks.d.ts +39 -44
  32. package/dist/config/esm-hooks.js +238 -63
  33. package/dist/config/load.d.ts +1 -7
  34. package/dist/config/load.js +23 -66
  35. package/dist/config/register.d.ts +8 -0
  36. package/dist/config/register.js +9 -0
  37. package/dist/config/specifiers.d.ts +19 -0
  38. package/dist/config/specifiers.js +33 -0
  39. package/dist/config/tsconfig.d.ts +56 -0
  40. package/dist/config/tsconfig.js +115 -0
  41. package/dist/config/typescript.d.ts +19 -0
  42. package/dist/config/typescript.js +253 -0
  43. package/dist/engine/contract.d.ts +1 -0
  44. package/dist/engine/index.d.ts +2 -0
  45. package/dist/engine/session.js +2 -2
  46. package/dist/expect/async.js +2 -1
  47. package/dist/expect/poll.js +2 -2
  48. package/dist/internal/errors.d.ts +10 -3
  49. package/dist/internal/errors.js +23 -8
  50. package/dist/internal/node-version.d.ts +16 -7
  51. package/dist/internal/node-version.js +50 -15
  52. package/dist/internal/package-manager.js +1 -1
  53. package/dist/internal/realm-slot.d.ts +5 -4
  54. package/dist/internal/redact.d.ts +2 -0
  55. package/dist/internal/redact.js +4 -0
  56. package/dist/internal/time.d.ts +21 -5
  57. package/dist/internal/time.js +35 -3
  58. package/dist/locator/engine.js +47 -11
  59. package/dist/locator/screen.js +31 -5
  60. package/dist/mcp/output.d.ts +59 -0
  61. package/dist/mcp/output.js +129 -0
  62. package/dist/mcp/server.d.ts +4 -0
  63. package/dist/mcp/server.js +9 -12
  64. package/dist/mcp/session.d.ts +8 -1
  65. package/dist/mcp/session.js +55 -14
  66. package/dist/mcp/tools.js +7 -5
  67. package/dist/report/list.d.ts +11 -0
  68. package/dist/report/list.js +23 -6
  69. package/dist/run/artifacts.d.ts +4 -4
  70. package/dist/run/artifacts.js +3 -2
  71. package/dist/run/execute.d.ts +25 -5
  72. package/dist/run/execute.js +44 -20
  73. package/dist/run/failure-evidence.js +6 -2
  74. package/dist/run/in-process.js +51 -10
  75. package/dist/run/process-output.d.ts +73 -0
  76. package/dist/run/process-output.js +130 -0
  77. package/dist/run/provision.d.ts +2 -0
  78. package/dist/run/provision.js +1 -0
  79. package/dist/run/records.d.ts +4 -4
  80. package/dist/run/retry.d.ts +7 -0
  81. package/dist/run/retry.js +27 -11
  82. package/dist/run/runner.d.ts +8 -0
  83. package/dist/run/runner.js +32 -9
  84. package/dist/run/scheduler.js +255 -19
  85. package/dist/run/secrecy.d.ts +22 -19
  86. package/dist/run/secrecy.js +22 -18
  87. package/dist/run/serial.d.ts +73 -2
  88. package/dist/run/serial.js +239 -58
  89. package/dist/run/standalone.js +1 -1
  90. package/dist/run/trace-redaction.d.ts +5 -6
  91. package/dist/run/trace-redaction.js +5 -6
  92. package/dist/run/worker/entry.js +35 -14
  93. package/dist/run/worker/handle.js +10 -5
  94. package/dist/run/worker/protocol.d.ts +83 -3
  95. package/dist/run/worker/session.js +28 -0
  96. package/dist/telemetry/environment.d.ts +2 -1
  97. package/dist/telemetry/environment.js +12 -2
  98. package/dist/telemetry/project.d.ts +5 -6
  99. package/dist/telemetry/project.js +6 -11
  100. package/dist/telemetry/store.d.ts +5 -5
  101. package/dist/telemetry/store.js +5 -7
  102. package/dist/telemetry/telemetry.d.ts +2 -3
  103. package/dist/telemetry/telemetry.js +2 -4
  104. package/dist/types.d.ts +6 -3
  105. package/docs/agent-steps.mdx +1 -0
  106. package/docs/agents.mdx +1 -0
  107. package/docs/api-testing.mdx +2 -1
  108. package/docs/authentication.mdx +3 -2
  109. package/docs/browser.mdx +7 -6
  110. package/docs/bug-bash.mdx +2 -1
  111. package/docs/cache.mdx +2 -1
  112. package/docs/ci.mdx +3 -2
  113. package/docs/coding-agents.mdx +2 -1
  114. package/docs/core-concepts.mdx +208 -0
  115. package/docs/debugging.mdx +19 -9
  116. package/docs/decision-models.mdx +142 -0
  117. package/docs/example-projects.mdx +32 -0
  118. package/docs/executors.mdx +3 -0
  119. package/docs/explore.mdx +1 -0
  120. package/docs/github.mdx +2 -1
  121. package/docs/index.mdx +9 -5
  122. package/docs/integrations/eas.mdx +1 -0
  123. package/docs/integrations/index.mdx +1 -0
  124. package/docs/integrations/kernel.mdx +2 -1
  125. package/docs/migrate/cypress.mdx +26 -3
  126. package/docs/migrate/detox.mdx +24 -1
  127. package/docs/migrate/maestro.mdx +28 -4
  128. package/docs/migrate/playwright.mdx +27 -4
  129. package/docs/migrate/selenium.mdx +27 -3
  130. package/docs/mobile-ci.mdx +6 -5
  131. package/docs/mobile.mdx +79 -8
  132. package/docs/models.mdx +7 -2
  133. package/docs/quickstart.mdx +26 -15
  134. package/docs/reference/agent.mdx +13 -3
  135. package/docs/reference/app.mdx +1 -0
  136. package/docs/reference/cli.mdx +11 -5
  137. package/docs/reference/config.mdx +42 -6
  138. package/docs/reference/engine.mdx +22 -10
  139. package/docs/reference/environment.mdx +2 -1
  140. package/docs/reference/errors.mdx +11 -8
  141. package/docs/reference/expect.mdx +2 -1
  142. package/docs/reference/mcp.mdx +21 -8
  143. package/docs/reference/mobile.mdx +1 -0
  144. package/docs/reference/reporters.mdx +1 -0
  145. package/docs/reference/screen.mdx +19 -9
  146. package/docs/reference/test.mdx +21 -1
  147. package/docs/reference/web.mdx +8 -2
  148. package/docs/security.mdx +33 -27
  149. package/docs/subscriptions.mdx +1 -0
  150. package/docs/telemetry.mdx +8 -5
  151. package/docs/tools.mdx +1 -0
  152. package/docs/{starting-your-app.mdx → web.mdx} +157 -11
  153. package/docs/writing-an-engine.mdx +2 -1
  154. package/package.json +11 -7
  155. package/skills/e2e/SKILL.md +3 -1
  156. package/skills/e2e/references/agent.md +11 -0
  157. package/skills/e2e/references/debugging.md +1 -1
  158. package/skills/e2e/references/mcp.md +8 -3
  159. package/skills/e2e/references/running.md +2 -1
  160. package/skills/e2e/references/setup.md +9 -1
  161. package/skills/e2e/references/writing-tests.md +1 -1
  162. package/docs/assertions.mdx +0 -94
  163. package/docs/goals.mdx +0 -176
  164. package/docs/locators.mdx +0 -64
  165. package/docs/physical-devices.mdx +0 -76
  166. package/docs/viewports.mdx +0 -146
@@ -0,0 +1,208 @@
1
+ ---
2
+ title: Core concepts
3
+ description: Goals, assertions, and locators, and how to structure and run tests with them.
4
+ icon: "lightbulb"
5
+ ---
6
+
7
+ A test has three types of step:
8
+
9
+ | Step | API | Model call |
10
+ | --- | --- | --- |
11
+ | [Goal](#goals) | `agent.act` | Yes, unless the [cache](/cache) replays it |
12
+ | [Assertion](#assertions) | `agent.assert`, `agent.waitFor`, `agent.extract` | Yes |
13
+ | [Locator](#locators) | `screen` and `expect` | No |
14
+
15
+ The model column is for the built-in agent. A
16
+ [custom executor](/executors) can run `act` and `assert` without a model.
17
+
18
+ You can use all three in one test:
19
+
20
+ ```ts title="tests/todos.e2e.ts"
21
+ import { test, expect } from 'e2e';
22
+
23
+ test('todos survive a filter round-trip', async ({ app, agent, screen }) => {
24
+ await app.open('/todos');
25
+
26
+ await agent.act('add todos "Buy milk" and "Walk the dog", complete the first');
27
+ await agent.assert('only one todo remains open');
28
+
29
+ await screen.getByRole('tab', 'Done').tap();
30
+ await expect(screen.getByTestId('todo')).toHaveCount(1);
31
+ });
32
+ ```
33
+
34
+ Await each agent call, screen call, and `expect.poll`. If a test returns
35
+ while a step still runs, the test fails with `STEP_NOT_AWAITED`.
36
+
37
+ ## Goals
38
+
39
+ `agent.act` takes one goal. The agent selects and does the actions. It
40
+ returns when the goal is complete. It throws when the goal fails.
41
+
42
+ ```ts
43
+ await agent.act('complete checkout with the test card');
44
+ await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
45
+ ```
46
+
47
+ - Write one goal for each call. The agent controls the flow in a goal. You
48
+ control the order of the goals.
49
+ - Use the words that the screen shows.
50
+ - Give test data in `params`. For a password, give a `Secret`. The model
51
+ does not see a secret. See [Signing in](/authentication).
52
+ - For a value that changes on each run, use `unique()`. Then the
53
+ [replay cache](/cache#what-must-match) can replay the step:
54
+
55
+ ```ts
56
+ import { unique } from 'e2e';
57
+
58
+ const email = `ada+${Date.now()}@example.test`;
59
+ await agent.act('sign up with email {email}', { params: { email: unique(email) } });
60
+ ```
61
+
62
+ If a goal fails, read the report. Then make the instruction more specific.
63
+ Put words that many goals use in the agent's `context`; see
64
+ [Agents and personas](/agents). For verdicts and exit codes, see
65
+ [How agent steps work](/agent-steps#how-a-step-ends).
66
+
67
+ `act` gets screenshots when it needs them. On some engines, it can tap and
68
+ type at points in the image.
69
+
70
+ ## Assertions
71
+
72
+ An assertion examines the screen. The judging model sees the statement, the
73
+ current screen, and the agent's context. It does not see the earlier steps
74
+ or the summaries of `act`.
75
+
76
+ - `agent.assert` examines the screen now. A false statement fails with
77
+ `ASSERTION_FAILED`. Not sufficient evidence fails with
78
+ `ASSERTION_INCONCLUSIVE`.
79
+ - `agent.waitFor` waits until a condition is true. It calls the model only
80
+ when the screen changes.
81
+ - `agent.extract` reads data from the screen into a
82
+ [Standard Schema](https://standardschema.dev) type, for example Zod.
83
+
84
+ ```ts
85
+ import { expect } from 'e2e';
86
+ import { z } from 'zod';
87
+
88
+ await agent.assert('the dashboard shows a trial badge');
89
+ await agent.waitFor('a download link appears', { timeout: 120_000 });
90
+
91
+ const data = await agent.extract('every todo title', {
92
+ schema: z.object({ titles: z.array(z.string()) }),
93
+ });
94
+ expect(data.titles).toContain('Buy milk');
95
+ ```
96
+
97
+ For text that the model wrote, examine the meaning, not the full sentence.
98
+ When the exact text is important, use a [locator](#locators).
99
+
100
+ ### Judging pixels
101
+
102
+ Judgments read a redacted text snapshot of the screen. For a visual check,
103
+ such as a chart or a layout, add `vision: true`. To judge only the
104
+ screenshot, use `vision: 'only'`.
105
+
106
+ ```ts
107
+ await agent.assert('the chart trends upward', { vision: true });
108
+ ```
109
+
110
+ Images add input tokens. After a secret fill, the attempt sends no more
111
+ screenshots. See the [agent reference](/reference/agent#vision).
112
+
113
+ ## Locators
114
+
115
+ A locator finds a known control. Locator steps do not call a model. Use them
116
+ when you know the exact action or the exact value.
117
+
118
+ ```ts
119
+ // Role and name first, label second, test ID last.
120
+ await screen.getByRole('button', 'Sign in').tap();
121
+ await screen.getByLabel('Email').fill('ada@example.test');
122
+
123
+ // Scope into a row.
124
+ const row = screen.getByRole('listitem').filter({ hasText: 'Design review' });
125
+ await row.getByRole('button', 'Archive').tap();
126
+
127
+ // Gestures and coordinates.
128
+ await screen.swipe({ direction: 'up' });
129
+ await screen.getByRole('image', 'Seat map').tap({ position: { x: 120, y: 48 } });
130
+
131
+ // Assertions retry until they pass or time out.
132
+ await expect(screen.getByRole('status')).toHaveText('2 remaining');
133
+ ```
134
+
135
+ - An action, a read, or a single-node assertion fails when its query finds
136
+ more than one node. Use `.filter()`, `.first()`, or `.nth()` to find one
137
+ node.
138
+ - A direct read, such as `textContent()`, does not retry.
139
+ - Put an `expect` after an `agent.act` goal. It examines the exact result.
140
+ The [replay cache](/cache) records a goal only after a later check passes.
141
+
142
+ See the [screen reference](/reference/screen) and the
143
+ [expect reference](/reference/expect#locator-matchers). For routes, cookies,
144
+ dialogs, and downloads, see [Working with the browser](/browser).
145
+
146
+ ## Structure
147
+
148
+ ```ts
149
+ import { beforeEach, describe, test } from 'e2e';
150
+
151
+ describe('billing', { tags: ['billing'] }, () => {
152
+ beforeEach(async ({ app }) => { await app.open('/billing'); });
153
+
154
+ test('upgrades', { retries: 2, timeout: 60_000 }, async ({ agent }) => {
155
+ await agent.act('upgrade to Pro');
156
+ });
157
+ });
158
+
159
+ test('needs two organizations', async ({ app }) => {
160
+ test.skip((await countOrganizations()) < 2, 'nothing to switch to');
161
+ await app.open('/organizations');
162
+ });
163
+ ```
164
+
165
+ - Each test runs in its own attempt. Prepare its data in fixtures or hooks.
166
+ - `test.skip(condition, reason)` stops the test when the condition is true.
167
+ The report shows the test as `skipped`.
168
+ - For a flow across many tests, set `{ serial: true }` on the group. The
169
+ tests run in sequence on one worker, and retry together.
170
+ - For setup and cleanup, use `test.extend`. See
171
+ [Your own fixtures](/reference/test#your-own-fixtures).
172
+ - Import `test` from `@e2e-dev/web` or `@e2e-dev/mobile` to have the
173
+ `browser` or `device` fixture typed.
174
+
175
+ The [test reference](/reference/test) shows all options.
176
+
177
+ ## Run one test
178
+
179
+ <CodeGroup>
180
+ ```bash npm
181
+ npx e2e run tests/signup.e2e.ts:12 # the test at line 12
182
+ npx e2e run --grep checkout # tests with a matching title
183
+ npx e2e run --target web # one target
184
+ ```
185
+
186
+ ```bash pnpm
187
+ pnpm exec e2e run tests/signup.e2e.ts:12 # the test at line 12
188
+ pnpm exec e2e run --grep checkout # tests with a matching title
189
+ pnpm exec e2e run --target web # one target
190
+ ```
191
+
192
+ ```bash bun
193
+ bunx e2e run tests/signup.e2e.ts:12 # the test at line 12
194
+ bunx e2e run --grep checkout # tests with a matching title
195
+ bunx e2e run --target web # one target
196
+ ```
197
+ </CodeGroup>
198
+
199
+ The [CLI reference](/reference/cli#e2e-run) shows all flags.
200
+
201
+ <CardGroup cols={2}>
202
+ <Card title="Caching" icon="box-archive" href="/cache">
203
+ Replay passed goals with no model calls.
204
+ </Card>
205
+ <Card title="How agent steps work" icon="list-check" href="/agent-steps">
206
+ Verdicts, exit codes, and what the model sees.
207
+ </Card>
208
+ </CardGroup>
@@ -1,6 +1,7 @@
1
1
  ---
2
2
  title: Debugging a run
3
3
  description: Read the report, open the evidence, watch the browser, and see what the model was sent.
4
+ icon: "magnifying-glass"
4
5
  ---
5
6
 
6
7
  Start with the failure in the terminal. Every run also writes
@@ -63,7 +64,9 @@ URL: `http://127.0.0.1:3000/todos`
63
64
 
64
65
  `Look at` points to the failing call. Assertion failures include expected
65
66
  and observed values; locator failures include the requested locator and
66
- nearby matches. Repeated failure details help you compare attempts.
67
+ nearby matches. Repeated failure details help you compare attempts. The
68
+ runner spends at most 5 s capturing the screen and screenshot once per failed
69
+ attempt; an app that does not answer in time gets no screen section.
67
70
 
68
71
  ## The report
69
72
 
@@ -105,9 +108,9 @@ reruns names, and writes its own under `rerun-<n>/`:
105
108
  `--trace off` skips the cost for one run, `--trace` records every attempt
106
109
  of a CI run.
107
110
  - **Downloads** saved through `browser.waitForDownload`, recorded with kind
108
- `download`. They are the app's bytes: `redaction: "incomplete"` unless a
109
- secret reached the session (filled, or held by the engine for basic auth)
110
- and the file is text, which the runner rewrites and marks `complete`.
111
+ `download`. They are the app's bytes: `redaction: "incomplete"` unless the
112
+ session knows a secret value and the file is text, which the runner rewrites and
113
+ marks `complete`.
111
114
  - **Videos** and **agent transcripts** when you ask for them; see below.
112
115
 
113
116
  After a secret fill, no screenshots are taken for the rest of the attempt; see
@@ -196,9 +199,16 @@ stdout | web tests/todos.e2e.ts > adds a todo
196
199
  hello from the test
197
200
  ```
198
201
 
199
- Nothing is buffered or dropped: every line appears when it is written, in a
200
- pipe or a CI log as much as in a terminal. The output is not part of the
201
- report; use `expect` or an artifact for anything a later reader needs.
202
+ Output from outside a test, such as a test file's top level or a reporter,
203
+ prints under `stdout | runner` or `stderr | runner`. Both streams print on
204
+ stdout, in a pipe or a CI log too. Without the `list` reporter, output
205
+ goes to its own stream as written.
206
+
207
+ Secret values are [redacted](/security#redaction). Output printed while the
208
+ config loads appears once it has loaded, before the run header, and is
209
+ dropped for a warning if the load fails. Every other line appears when it
210
+ is written. The output is not part of the report; use `expect` or an
211
+ artifact for anything a later reader needs.
202
212
 
203
213
  ## Agent steps
204
214
 
@@ -264,13 +274,13 @@ large raw JSON file.
264
274
 
265
275
  Replayed steps make no model call and leave no AI trace. Add `--no-cache`
266
276
  when you need to inspect the whole flow or rule out a stale recording. See
267
- [Caching agent steps](/cache).
277
+ [Caching](/cache).
268
278
 
269
279
  ## Common errors
270
280
 
271
281
  | Code | Usual cause | Fix |
272
282
  | --- | --- | --- |
273
- | `APP_UNREACHABLE` | Nothing answers at `app.url`, or `app.command` never became ready; on a device, the message may name the iOS automation runner instead (busy, wedged, or a snapshot it could not present) | Set `app.command.log` and read it; check the port; raise `app.command.startupTimeout`. For the runner, follow the recovery in the message: `npx agent-device daemon stop`, reboot the simulator (see [Testing iOS and Android](/mobile#troubleshooting)) |
283
+ | `APP_UNREACHABLE` | Nothing answers at `app.url`, or `app.command` never became ready; on a device, the message may name the iOS automation runner instead (busy, wedged, or a snapshot it could not present) | Set `app.command.log` and read it; check the port; raise `app.command.startupTimeout`. For the runner, follow the recovery in the message: `npx agent-device daemon stop`, reboot the simulator (see [Mobile](/mobile#troubleshooting)) |
274
284
  | `APP_ALREADY_RUNNING` | Something already serves `app.url` when the runner wanted to start `app.command` | Stop it, or set `reuseExisting: true` for local runs |
275
285
  | `LOCATOR_NOT_FOUND` | Wrong role or name, text not exact, element inside an iframe, page not open | Read the markup for the accessible name; try `exact: false`; `browser.frameLocator` for iframes; `--headed` to look |
276
286
  | `LOCATOR_AMBIGUOUS` | Two matches: a hidden duplicate, a repeated label | Add `{ name }`, scope under a container, `filter`, `first()`, or `{ visible: true }` |
@@ -0,0 +1,142 @@
1
+ ---
2
+ title: Decision models
3
+ description: Run bounded actions with a decision model and a small text model instead of a full LLM agent.
4
+ icon: "scale-balanced"
5
+ ---
6
+
7
+ `@e2e-dev/decision` runs `agent.act` and `agent.assert` through a decision
8
+ model instead of an LLM. For each action the decision model answers one
9
+ request: which operation, and which element for each operation that has
10
+ more than one, chosen from what the executor builds out of the latest
11
+ semantic tree. It never generates free text: when the choice is `type`, a
12
+ small language model writes the field value. Both models are
13
+ plain AI SDK models, so any decision model provider that answers `choice`
14
+ questions with probability distributions (Jev) and any language-model
15
+ provider works; the runner still authorizes every dispatched action.
16
+
17
+ ## Install and configure
18
+
19
+ <CodeGroup>
20
+ ```bash npm
21
+ npm install -D @e2e-dev/decision
22
+ ```
23
+
24
+ ```bash pnpm
25
+ pnpm add -D @e2e-dev/decision
26
+ ```
27
+
28
+ ```bash bun
29
+ bun add -d @e2e-dev/decision
30
+ ```
31
+ </CodeGroup>
32
+
33
+ The decision model answers `choice` questions with probability
34
+ distributions; the text model writes field values. Each installs as its
35
+ own package. Decision models need `ai` 7, version 7.0.128 or later:
36
+
37
+ <CodeGroup>
38
+ ```bash npm
39
+ npm install -D @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider
40
+ ```
41
+
42
+ ```bash pnpm
43
+ pnpm add -D @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider
44
+ ```
45
+
46
+ ```bash bun
47
+ bun add -d @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider
48
+ ```
49
+ </CodeGroup>
50
+
51
+ ```ts title="e2e.config.ts"
52
+ import type { E2EConfig } from 'e2e';
53
+ import { web } from '@e2e-dev/web';
54
+ import { decisionExecutor } from '@e2e-dev/decision';
55
+ import { typeSafeAi } from '@ai-sdk/typesafe-ai';
56
+ import { openrouter } from '@openrouter/ai-sdk-provider';
57
+
58
+ export default {
59
+ targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
60
+ agents: {
61
+ default: {
62
+ executor: decisionExecutor({
63
+ model: typeSafeAi.decisionModel('jev-latest'),
64
+ textModel: openrouter('inception/mercury-2.5'),
65
+ }),
66
+ },
67
+ },
68
+ } satisfies E2EConfig;
69
+ ```
70
+
71
+ `evaluationModel()` instances still work. A model id string or a language
72
+ model in `model` fails config load with `INVALID_CONFIG`.
73
+
74
+ Tests stay plain natural language, with no params needed:
75
+
76
+ ```ts title="tests/todos.e2e.ts"
77
+ import { test, expect } from 'e2e';
78
+
79
+ test('adds a todo', async ({ app, screen, agent }) => {
80
+ await app.open('/todos');
81
+ await agent.act('Add a todo named Buy coffee, and verify it is listed.');
82
+ await expect(screen.getByTestId('todo')).toContainText('Buy coffee');
83
+ await agent.assert('The todo list contains Buy coffee.');
84
+ });
85
+ ```
86
+
87
+ ## How text gets typed
88
+
89
+ The decision model only ever names an element index. When it picks `type`,
90
+ the text model writes the value from the goal, the field's label, and the
91
+ page text, and the executor types exactly that string. Model output never
92
+ becomes a selector, a URL, or a key. When the goal supplies no value,
93
+ nothing is typed and the decision model picks another action. Password
94
+ fields never reach the text model; secrets stay declared handles that only
95
+ `typeSecret` fills.
96
+
97
+ The text model is also the agent's judgment tier: it is exposed as the
98
+ executor's `model`, so `waitFor` and `extract` use it too. It can come
99
+ from the agents entry's `model` instead. Without either, `type` is never
100
+ offered.
101
+
102
+ ## Finishing a step
103
+
104
+ When the decision model answers `done` or `failed`, the executor checks
105
+ the claim on a fresh screen with one question: is the step's task complete?
106
+ It sees the step's non-secret params, the actions taken with the non-secret
107
+ values typed, and the page, including the text of named status regions, but
108
+ no model reasoning or secret. A confirmed `done` passes the
109
+ step and a confirmed `failed` fails it with `ACTION_FAILED`. An
110
+ unconfirmed claim goes back to the decision model as history; the second one
111
+ ends the step, as `ACTION_FAILED` when the screen contradicts the goal and
112
+ as `blocked` (`AUTOMATION_UNSUPPORTED`) otherwise.
113
+
114
+ ## Gates
115
+
116
+ Both gates are off by default: the executor follows the most probable
117
+ choice. Set `minProbability` or `minConfidence` to block weaker actions,
118
+ reject weaker completion checks, and fail weaker assertions as
119
+ inconclusive instead:
120
+
121
+ ```ts title="Executor setup"
122
+ const executor = decisionExecutor({ model, minProbability: 0.9, minConfidence: 0.8 });
123
+ ```
124
+
125
+ Probability and confidence are different provider fields: probability is the
126
+ selected option's share of the answer distribution, confidence is the
127
+ provider's separate statistic. A threshold is a policy choice; measure
128
+ accuracy on your own tests before changing it.
129
+
130
+ ## Limits
131
+
132
+ - No vision: screenshots never reach the model; canvas-only controls and
133
+ image-based targets are not implemented.
134
+ - No `navigate`: in-app navigation is a `tap` on a link. URLs belong in
135
+ test code (`browser.goto`, `app.open`).
136
+ - No multi-select `listbox` in v1.
137
+ - The choice cap is 255 options per question (TypeSafe's limit). Larger
138
+ screens list viewport nodes first and count the rest as omitted, so the
139
+ model can scroll; the step never blocks just for being large.
140
+ - Drag, upload, and hover are not offered in v1.
141
+ - `waitFor` and `extract` use the text model through the runner's
142
+ built-in judgment tier.
@@ -0,0 +1,32 @@
1
+ ---
2
+ title: Example projects
3
+ description: Run standalone e2e examples with Vite, Next.js, Expo, and SwiftUI.
4
+ icon: "folder-open"
5
+ ---
6
+
7
+ Each example is a standalone greeting app with three locator tests and two
8
+ agent tests. Enter a name, press Greet, and check the result. The projects
9
+ install published e2e packages from npm and work outside this repository.
10
+
11
+ <CardGroup cols={2}>
12
+ <Card title="Vite" icon="/images/examples/vite.svg" href="https://github.com/tester-army/e2e/tree/main/examples/with-vite" cta="View example" arrow>
13
+ React with Vite. Run locator and agent tests in Chromium.
14
+ </Card>
15
+ <Card title="Next.js" icon="/images/examples/nextjs.svg" href="https://github.com/tester-army/e2e/tree/main/examples/with-next" cta="View example" arrow>
16
+ Next.js App Router. The runner starts the app and tests it in Chromium.
17
+ </Card>
18
+ <Card title="Expo" icon="/images/integrations/expo.svg" href="https://github.com/tester-army/e2e/tree/main/examples/with-expo" cta="View example" arrow>
19
+ React Native with Expo. Run the same tests on web, an iOS simulator, or an Android emulator.
20
+ </Card>
21
+ <Card title="SwiftUI" icon="/images/examples/swiftui.png" href="https://github.com/tester-army/e2e/tree/main/examples/with-swiftui" cta="View example" arrow>
22
+ A native iOS app with an Xcode project. Run tests on an iPhone simulator.
23
+ </Card>
24
+ </CardGroup>
25
+
26
+ Follow each project's README for installation, builds, and run commands.
27
+ The config and tests live beside the app. The SwiftUI project keeps them
28
+ in its `e2e/` folder.
29
+
30
+ Locator tests need no model. Set `AI_GATEWAY_API_KEY` to run the agent
31
+ tests; without it, those tests skip. See [Models](/models) for model setup
32
+ and the [Quickstart](/quickstart) to add e2e to your own app.
@@ -1,6 +1,7 @@
1
1
  ---
2
2
  title: Custom executors
3
3
  description: Customize the agent loop or implement your own step executor.
4
+ icon: "microchip"
4
5
  ---
5
6
 
6
7
  Use a custom executor when a `system` prompt and [project tools](/tools) are
@@ -14,6 +15,8 @@ not enough. There are two options:
14
15
  `ai` package installed.
15
16
 
16
17
  Both receive a deadline, redacted observations, and checked action methods.
18
+ For a provider that returns bounded choices instead of generated text, see
19
+ the [Clef and Jev decision executor](/decision-models).
17
20
  The runner records their verdicts. Either goes under `executor` in an agents
18
21
  entry, beside the options that still apply: `model`, `judge`, `context`, and
19
22
  the budgets. An entry with `executor` rejects `system` and `tools`; pass
package/docs/explore.mdx CHANGED
@@ -1,6 +1,7 @@
1
1
  ---
2
2
  title: Exploring without a test
3
3
  description: Give the agent a goal instead of a test file. It plans its own steps, drives the app, and reports what it finds.
4
+ icon: "compass"
4
5
  ---
5
6
 
6
7
  Use `e2e explore` to investigate a flow without writing a test. Give the
package/docs/github.mdx CHANGED
@@ -1,6 +1,7 @@
1
1
  ---
2
2
  title: Pull request comments
3
3
  description: The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.
4
+ icon: "code-pull-request"
4
5
  ---
5
6
 
6
7
  `@e2e-dev/github` posts test results as a pull request comment, including
@@ -73,7 +74,7 @@ that path. The checkout root is the nearest directory holding `.git` inside
73
74
 
74
75
  ## The comment
75
76
 
76
- ```markdown
77
+ ```markdown expandable
77
78
  ### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
78
79
  9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31
79
80
 
package/docs/index.mdx CHANGED
@@ -1,6 +1,7 @@
1
1
  ---
2
2
  title: Introduction
3
3
  description: Open source AI testing for web and mobile apps. Spend less time on tests and more time building.
4
+ icon: "house"
4
5
  ---
5
6
 
6
7
  import { VideoCard } from './snippets/video-card.jsx';
@@ -9,9 +10,9 @@ e2e is a next-generation testing framework for web and mobile
9
10
  apps, built to keep up with the speed we ship software today.
10
11
 
11
12
  You choose how agentic each test is: fully deterministic with
12
- [locators](/locators) and
13
+ [locators](/core-concepts#locators) and
13
14
  [assertions](/reference/expect#locator-matchers), fully driven by an agent
14
- from [goals](/goals) in natural language, or a mix of
15
+ from [goals](/core-concepts#goals) in natural language, or a mix of
15
16
  both in the same test.
16
17
 
17
18
  <VideoCard
@@ -49,7 +50,7 @@ you shipped.
49
50
  ## How it works
50
51
 
51
52
  - **Targets.** Each test runs once per target, such as Chromium or an iPhone
52
- simulator. The [web engine](/starting-your-app) drives browsers through
53
+ simulator. The [web engine](/web) drives browsers through
53
54
  Playwright, the [mobile engine](/mobile) drives iOS simulators and Android
54
55
  emulators through agent-device, and you can
55
56
  [write your own](/writing-an-engine).
@@ -70,8 +71,11 @@ you shipped.
70
71
  <Card title="Quickstart" icon="rocket" href="/quickstart">
71
72
  One prompt for your coding agent, or set up by hand.
72
73
  </Card>
73
- <Card title="Goals" icon="pen-to-square" href="/goals">
74
- Write test steps as goals in natural language.
74
+ <Card title="Example projects" icon="folder-open" href="/example-projects">
75
+ Run a standalone app with Vite, Next.js, Expo, or SwiftUI.
76
+ </Card>
77
+ <Card title="Core concepts" icon="pen-to-square" href="/core-concepts">
78
+ Goals, assertions, and locators.
75
79
  </Card>
76
80
  <Card title="Coding agents" icon="robot" href="/coding-agents">
77
81
  Let your coding agent write, run, and debug tests.
@@ -2,6 +2,7 @@
2
2
  title: EAS Simulators
3
3
  sidebarTitle: EAS Simulators
4
4
  description: Run mobile targets on EAS Simulators' hosted iOS simulators and Android emulators with the easSimulators() device provider from @e2e-dev/eas.
5
+ icon: "/images/icons/expo.svg"
5
6
  ---
6
7
 
7
8
  `easSimulators()` from `@e2e-dev/eas` is a
@@ -2,6 +2,7 @@
2
2
  title: Integrations
3
3
  sidebarTitle: Overview
4
4
  description: Official integrations with hosted services, one package per service.
5
+ icon: "plug"
5
6
  ---
6
7
 
7
8
  Official integrations connect e2e to hosted services through the engines'
@@ -2,6 +2,7 @@
2
2
  title: Kernel browsers
3
3
  sidebarTitle: Kernel
4
4
  description: Run web targets in Kernel's hosted Chromium with the kernel() browser provider from @e2e-dev/kernel.
5
+ icon: "/images/icons/kernel.svg"
5
6
  ---
6
7
 
7
8
  `kernel()` from `@e2e-dev/kernel` is a
@@ -60,7 +61,7 @@ web({
60
61
  ```
61
62
 
62
63
  Each attempt's window then opens maximized, and the page
63
- [fills it](/viewports#fill-the-window): 1920 by 993 on a 1920 by 1080 screen,
64
+ [fills it](/web#fill-the-window): 1920 by 993 on a 1920 by 1080 screen,
64
65
  after the tab strip and address bar. With a fixed web `viewport`, the page is
65
66
  emulated at that size inside a smaller window, so Kernel's live view shows it
66
67
  cut off or in part of the window.
@@ -1,8 +1,31 @@
1
1
  ---
2
2
  title: From Cypress
3
3
  description: Replace Cypress command chains with async tests and keep your data-cy attributes.
4
+ icon: "/images/icons/cypress.svg"
4
5
  ---
5
6
 
7
+ import { SetupPrompt } from '../snippets/setup-prompt-card.jsx';
8
+
9
+ export const migrationPrompt = `Migrate this project's Cypress tests to e2e, one spec at a time. Docs: \`https://e2e.tester.army/docs/migrate/cypress.md\`
10
+
11
+ 1. Read the migration guide above. It maps Cypress config, commands, and assertions to e2e.
12
+ 2. If the project has no e2e config (\`e2e.config.ts\` or \`e2e.config.mts\`), run \`npx e2e init --yes\` (or the pnpm or bun equivalent). It writes a config and an example test, and installs nothing yet.
13
+ 3. Read the e2e skill (\`npx e2e guide\` prints it, \`npx e2e guide <topic>\` prints a topic) and follow it for every step below.
14
+ 4. If you ran init, ask me which model to use for agent steps: a ChatGPT, GitHub Copilot, OpenCode Console, or SuperGrok subscription, an API key, or a local model. Configure it with its provider package. If it needs a sign-in, give me the \`npx e2e login\` command and wait until I confirm.
15
+ 5. Port \`cypress.config.*\` into the e2e config: \`baseUrl\` and the dev server command become the target's \`app\`, and if the specs select by \`data-cy\`, set \`web({ testIdAttribute: 'data-cy' })\` so \`screen.getByTestId\` reads it. Keep Cypress installed while both suites run.
16
+ 6. Install dependencies.
17
+ 7. List the Cypress end-to-end specs and ask me which one to move first. Suggest one that breaks often or waits with \`cy.wait(ms)\`. Component tests have no e2e equivalent; leave them in Cypress.
18
+ 8. Rewrite that spec as a \`.e2e.ts\` file under \`tests/\` (e2e runs \`tests/**/*.e2e.ts\` by default), with one awaited call per line and no chains. \`cy.contains\` matches a substring and e2e queries match the whole text, so pass \`{ exact: false }\` where the spec relied on a fragment. Replace each \`cy.wait(ms)\` with an assertion on the state the test waits for. Move \`cy.task\`, \`cy.exec\`, and \`cy.request\` into plain Node code, and custom commands into functions or fixtures.
19
+ 9. Keep navigation and the final assertions deterministic, with \`app.open()\` and locator assertions. Where a run of scripted interactions breaks often, propose an \`agent.act\` goal for it instead.
20
+ 10. Run the new file with \`npx e2e run <file>\` and fix it until it passes. Keep the Cypress spec until I confirm it can go.
21
+ 11. Summarize which tests moved, which stayed in Cypress, and why.`;
22
+
23
+ <SetupPrompt>
24
+
25
+ <Prompt description={migrationPrompt} children={migrationPrompt} actions={["copy", "cursor"]} />
26
+
27
+ </SetupPrompt>
28
+
6
29
  e2e tests run in Node. Use `await` for UI actions and call Node functions
7
30
  directly for work previously handled by `cy.task`. Both suites can stay in
8
31
  the project while you migrate one spec at a time.
@@ -15,7 +38,7 @@ for the agent calls shown below.
15
38
  | Cypress | e2e |
16
39
  | --- | --- |
17
40
  | `baseUrl` | `app: { url }` on the target |
18
- | `start-server-and-test` | `app: { url, command }`, see [Starting your app](/starting-your-app) |
41
+ | `start-server-and-test` | `app: { url, command }`, see [Starting your app](/web#starting-your-app) |
19
42
  | `viewportWidth`, `viewportHeight` | `web({ viewport })` |
20
43
  | `data-cy` selectors | `web({ testIdAttribute: 'data-cy' })`, then `screen.getByTestId` |
21
44
  | `retries` | `retries` |
@@ -115,8 +138,8 @@ Start with a spec that often breaks and whose fixed `cy.wait` calls can
115
138
  become assertions on the condition the test needs.
116
139
 
117
140
  <CardGroup cols={2}>
118
- <Card title="Goals" icon="pen-to-square" href="/goals">
119
- Write test steps as goals instead of scripted selectors.
141
+ <Card title="Core concepts" icon="pen-to-square" href="/core-concepts">
142
+ Goals, assertions, and locators instead of scripted selectors.
120
143
  </Card>
121
144
  <Card title="Working with the browser" icon="globe" href="/browser">
122
145
  Routes, dialogs, downloads, iframes, and page evaluation.