e2e 0.17.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -0
- package/dist/.build.json +1 -1
- package/dist/agent/ai-sdk.js +5 -4
- package/dist/agent/error.js +3 -3
- package/dist/agent/invocation.js +3 -4
- package/dist/agent/model/tool-choice.d.ts +9 -0
- package/dist/agent/model/tool-choice.js +14 -0
- package/dist/agent/observation-feed.js +3 -4
- package/dist/agent/phases.d.ts +7 -1
- package/dist/agent/phases.js +12 -3
- package/dist/agent/prompts.js +3 -0
- package/dist/agent/replay.js +18 -15
- package/dist/agent/step-cache.d.ts +2 -1
- package/dist/agent/step-cache.js +9 -5
- package/dist/agent/tool-loop.js +92 -5
- package/dist/cache/trace.js +6 -2
- package/dist/cli/bin.js +2 -2
- package/dist/cli/index.js +35 -24
- package/dist/cli/init/mcp-config.d.ts +2 -1
- package/dist/cli/init/mcp-config.js +8 -3
- package/dist/cli/init/sibling-versions.json +1 -1
- package/dist/cli/mcp.d.ts +3 -2
- package/dist/cli/mcp.js +52 -16
- package/dist/cli/run-output.d.ts +17 -0
- package/dist/cli/run-output.js +45 -0
- package/dist/collect/collect.js +8 -4
- package/dist/collect/registry.js +27 -16
- package/dist/config/compiled-files.d.ts +36 -0
- package/dist/config/compiled-files.js +50 -0
- package/dist/config/diagnose.js +21 -19
- package/dist/config/esm-hooks.d.ts +39 -44
- package/dist/config/esm-hooks.js +238 -63
- package/dist/config/load.d.ts +1 -7
- package/dist/config/load.js +23 -66
- package/dist/config/register.d.ts +8 -0
- package/dist/config/register.js +9 -0
- package/dist/config/specifiers.d.ts +19 -0
- package/dist/config/specifiers.js +33 -0
- package/dist/config/tsconfig.d.ts +56 -0
- package/dist/config/tsconfig.js +115 -0
- package/dist/config/typescript.d.ts +19 -0
- package/dist/config/typescript.js +253 -0
- package/dist/engine/contract.d.ts +1 -0
- package/dist/engine/index.d.ts +2 -0
- package/dist/engine/session.js +2 -2
- package/dist/expect/async.js +2 -1
- package/dist/expect/poll.js +2 -2
- package/dist/internal/errors.d.ts +10 -3
- package/dist/internal/errors.js +23 -8
- package/dist/internal/node-version.d.ts +16 -7
- package/dist/internal/node-version.js +50 -15
- package/dist/internal/package-manager.js +1 -1
- package/dist/internal/realm-slot.d.ts +5 -4
- package/dist/internal/redact.d.ts +2 -0
- package/dist/internal/redact.js +4 -0
- package/dist/internal/time.d.ts +21 -5
- package/dist/internal/time.js +35 -3
- package/dist/locator/engine.js +47 -11
- package/dist/locator/screen.js +31 -5
- package/dist/mcp/output.d.ts +59 -0
- package/dist/mcp/output.js +129 -0
- package/dist/mcp/server.d.ts +4 -0
- package/dist/mcp/server.js +9 -12
- package/dist/mcp/session.d.ts +8 -1
- package/dist/mcp/session.js +55 -14
- package/dist/mcp/tools.js +7 -5
- package/dist/report/list.d.ts +11 -0
- package/dist/report/list.js +23 -6
- package/dist/run/artifacts.d.ts +4 -4
- package/dist/run/artifacts.js +3 -2
- package/dist/run/execute.d.ts +25 -5
- package/dist/run/execute.js +44 -20
- package/dist/run/failure-evidence.js +6 -2
- package/dist/run/in-process.js +51 -10
- package/dist/run/process-output.d.ts +73 -0
- package/dist/run/process-output.js +130 -0
- package/dist/run/provision.d.ts +2 -0
- package/dist/run/provision.js +1 -0
- package/dist/run/records.d.ts +4 -4
- package/dist/run/retry.d.ts +7 -0
- package/dist/run/retry.js +27 -11
- package/dist/run/runner.d.ts +8 -0
- package/dist/run/runner.js +32 -9
- package/dist/run/scheduler.js +255 -19
- package/dist/run/secrecy.d.ts +22 -19
- package/dist/run/secrecy.js +22 -18
- package/dist/run/serial.d.ts +73 -2
- package/dist/run/serial.js +239 -58
- package/dist/run/standalone.js +1 -1
- package/dist/run/trace-redaction.d.ts +5 -6
- package/dist/run/trace-redaction.js +5 -6
- package/dist/run/worker/entry.js +35 -14
- package/dist/run/worker/handle.js +10 -5
- package/dist/run/worker/protocol.d.ts +83 -3
- package/dist/run/worker/session.js +28 -0
- package/dist/telemetry/environment.d.ts +2 -1
- package/dist/telemetry/environment.js +12 -2
- package/dist/telemetry/project.d.ts +5 -6
- package/dist/telemetry/project.js +6 -11
- package/dist/telemetry/store.d.ts +5 -5
- package/dist/telemetry/store.js +5 -7
- package/dist/telemetry/telemetry.d.ts +2 -3
- package/dist/telemetry/telemetry.js +2 -4
- package/dist/types.d.ts +6 -3
- package/docs/agent-steps.mdx +1 -0
- package/docs/agents.mdx +1 -0
- package/docs/api-testing.mdx +2 -1
- package/docs/authentication.mdx +3 -2
- package/docs/browser.mdx +7 -6
- package/docs/bug-bash.mdx +2 -1
- package/docs/cache.mdx +2 -1
- package/docs/ci.mdx +3 -2
- package/docs/coding-agents.mdx +2 -1
- package/docs/core-concepts.mdx +208 -0
- package/docs/debugging.mdx +19 -9
- package/docs/decision-models.mdx +142 -0
- package/docs/example-projects.mdx +32 -0
- package/docs/executors.mdx +3 -0
- package/docs/explore.mdx +1 -0
- package/docs/github.mdx +2 -1
- package/docs/index.mdx +9 -5
- package/docs/integrations/eas.mdx +1 -0
- package/docs/integrations/index.mdx +1 -0
- package/docs/integrations/kernel.mdx +2 -1
- package/docs/migrate/cypress.mdx +26 -3
- package/docs/migrate/detox.mdx +24 -1
- package/docs/migrate/maestro.mdx +28 -4
- package/docs/migrate/playwright.mdx +27 -4
- package/docs/migrate/selenium.mdx +27 -3
- package/docs/mobile-ci.mdx +6 -5
- package/docs/mobile.mdx +79 -8
- package/docs/models.mdx +7 -2
- package/docs/quickstart.mdx +26 -15
- package/docs/reference/agent.mdx +13 -3
- package/docs/reference/app.mdx +1 -0
- package/docs/reference/cli.mdx +11 -5
- package/docs/reference/config.mdx +42 -6
- package/docs/reference/engine.mdx +22 -10
- package/docs/reference/environment.mdx +2 -1
- package/docs/reference/errors.mdx +11 -8
- package/docs/reference/expect.mdx +2 -1
- package/docs/reference/mcp.mdx +21 -8
- package/docs/reference/mobile.mdx +1 -0
- package/docs/reference/reporters.mdx +1 -0
- package/docs/reference/screen.mdx +19 -9
- package/docs/reference/test.mdx +21 -1
- package/docs/reference/web.mdx +8 -2
- package/docs/security.mdx +33 -27
- package/docs/subscriptions.mdx +1 -0
- package/docs/telemetry.mdx +8 -5
- package/docs/tools.mdx +1 -0
- package/docs/{starting-your-app.mdx → web.mdx} +157 -11
- package/docs/writing-an-engine.mdx +2 -1
- package/package.json +11 -7
- package/skills/e2e/SKILL.md +3 -1
- package/skills/e2e/references/agent.md +11 -0
- package/skills/e2e/references/debugging.md +1 -1
- package/skills/e2e/references/mcp.md +8 -3
- package/skills/e2e/references/running.md +2 -1
- package/skills/e2e/references/setup.md +9 -1
- package/skills/e2e/references/writing-tests.md +1 -1
- package/docs/assertions.mdx +0 -94
- package/docs/goals.mdx +0 -176
- package/docs/locators.mdx +0 -64
- package/docs/physical-devices.mdx +0 -76
- package/docs/viewports.mdx +0 -146
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Core concepts
|
|
3
|
+
description: Goals, assertions, and locators, and how to structure and run tests with them.
|
|
4
|
+
icon: "lightbulb"
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
A test has three types of step:
|
|
8
|
+
|
|
9
|
+
| Step | API | Model call |
|
|
10
|
+
| --- | --- | --- |
|
|
11
|
+
| [Goal](#goals) | `agent.act` | Yes, unless the [cache](/cache) replays it |
|
|
12
|
+
| [Assertion](#assertions) | `agent.assert`, `agent.waitFor`, `agent.extract` | Yes |
|
|
13
|
+
| [Locator](#locators) | `screen` and `expect` | No |
|
|
14
|
+
|
|
15
|
+
The model column is for the built-in agent. A
|
|
16
|
+
[custom executor](/executors) can run `act` and `assert` without a model.
|
|
17
|
+
|
|
18
|
+
You can use all three in one test:
|
|
19
|
+
|
|
20
|
+
```ts title="tests/todos.e2e.ts"
|
|
21
|
+
import { test, expect } from 'e2e';
|
|
22
|
+
|
|
23
|
+
test('todos survive a filter round-trip', async ({ app, agent, screen }) => {
|
|
24
|
+
await app.open('/todos');
|
|
25
|
+
|
|
26
|
+
await agent.act('add todos "Buy milk" and "Walk the dog", complete the first');
|
|
27
|
+
await agent.assert('only one todo remains open');
|
|
28
|
+
|
|
29
|
+
await screen.getByRole('tab', 'Done').tap();
|
|
30
|
+
await expect(screen.getByTestId('todo')).toHaveCount(1);
|
|
31
|
+
});
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Await each agent call, screen call, and `expect.poll`. If a test returns
|
|
35
|
+
while a step still runs, the test fails with `STEP_NOT_AWAITED`.
|
|
36
|
+
|
|
37
|
+
## Goals
|
|
38
|
+
|
|
39
|
+
`agent.act` takes one goal. The agent selects and does the actions. It
|
|
40
|
+
returns when the goal is complete. It throws when the goal fails.
|
|
41
|
+
|
|
42
|
+
```ts
|
|
43
|
+
await agent.act('complete checkout with the test card');
|
|
44
|
+
await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
- Write one goal for each call. The agent controls the flow in a goal. You
|
|
48
|
+
control the order of the goals.
|
|
49
|
+
- Use the words that the screen shows.
|
|
50
|
+
- Give test data in `params`. For a password, give a `Secret`. The model
|
|
51
|
+
does not see a secret. See [Signing in](/authentication).
|
|
52
|
+
- For a value that changes on each run, use `unique()`. Then the
|
|
53
|
+
[replay cache](/cache#what-must-match) can replay the step:
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
import { unique } from 'e2e';
|
|
57
|
+
|
|
58
|
+
const email = `ada+${Date.now()}@example.test`;
|
|
59
|
+
await agent.act('sign up with email {email}', { params: { email: unique(email) } });
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
If a goal fails, read the report. Then make the instruction more specific.
|
|
63
|
+
Put words that many goals use in the agent's `context`; see
|
|
64
|
+
[Agents and personas](/agents). For verdicts and exit codes, see
|
|
65
|
+
[How agent steps work](/agent-steps#how-a-step-ends).
|
|
66
|
+
|
|
67
|
+
`act` gets screenshots when it needs them. On some engines, it can tap and
|
|
68
|
+
type at points in the image.
|
|
69
|
+
|
|
70
|
+
## Assertions
|
|
71
|
+
|
|
72
|
+
An assertion examines the screen. The judging model sees the statement, the
|
|
73
|
+
current screen, and the agent's context. It does not see the earlier steps
|
|
74
|
+
or the summaries of `act`.
|
|
75
|
+
|
|
76
|
+
- `agent.assert` examines the screen now. A false statement fails with
|
|
77
|
+
`ASSERTION_FAILED`. Not sufficient evidence fails with
|
|
78
|
+
`ASSERTION_INCONCLUSIVE`.
|
|
79
|
+
- `agent.waitFor` waits until a condition is true. It calls the model only
|
|
80
|
+
when the screen changes.
|
|
81
|
+
- `agent.extract` reads data from the screen into a
|
|
82
|
+
[Standard Schema](https://standardschema.dev) type, for example Zod.
|
|
83
|
+
|
|
84
|
+
```ts
|
|
85
|
+
import { expect } from 'e2e';
|
|
86
|
+
import { z } from 'zod';
|
|
87
|
+
|
|
88
|
+
await agent.assert('the dashboard shows a trial badge');
|
|
89
|
+
await agent.waitFor('a download link appears', { timeout: 120_000 });
|
|
90
|
+
|
|
91
|
+
const data = await agent.extract('every todo title', {
|
|
92
|
+
schema: z.object({ titles: z.array(z.string()) }),
|
|
93
|
+
});
|
|
94
|
+
expect(data.titles).toContain('Buy milk');
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
For text that the model wrote, examine the meaning, not the full sentence.
|
|
98
|
+
When the exact text is important, use a [locator](#locators).
|
|
99
|
+
|
|
100
|
+
### Judging pixels
|
|
101
|
+
|
|
102
|
+
Judgments read a redacted text snapshot of the screen. For a visual check,
|
|
103
|
+
such as a chart or a layout, add `vision: true`. To judge only the
|
|
104
|
+
screenshot, use `vision: 'only'`.
|
|
105
|
+
|
|
106
|
+
```ts
|
|
107
|
+
await agent.assert('the chart trends upward', { vision: true });
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Images add input tokens. After a secret fill, the attempt sends no more
|
|
111
|
+
screenshots. See the [agent reference](/reference/agent#vision).
|
|
112
|
+
|
|
113
|
+
## Locators
|
|
114
|
+
|
|
115
|
+
A locator finds a known control. Locator steps do not call a model. Use them
|
|
116
|
+
when you know the exact action or the exact value.
|
|
117
|
+
|
|
118
|
+
```ts
|
|
119
|
+
// Role and name first, label second, test ID last.
|
|
120
|
+
await screen.getByRole('button', 'Sign in').tap();
|
|
121
|
+
await screen.getByLabel('Email').fill('ada@example.test');
|
|
122
|
+
|
|
123
|
+
// Scope into a row.
|
|
124
|
+
const row = screen.getByRole('listitem').filter({ hasText: 'Design review' });
|
|
125
|
+
await row.getByRole('button', 'Archive').tap();
|
|
126
|
+
|
|
127
|
+
// Gestures and coordinates.
|
|
128
|
+
await screen.swipe({ direction: 'up' });
|
|
129
|
+
await screen.getByRole('image', 'Seat map').tap({ position: { x: 120, y: 48 } });
|
|
130
|
+
|
|
131
|
+
// Assertions retry until they pass or time out.
|
|
132
|
+
await expect(screen.getByRole('status')).toHaveText('2 remaining');
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
- An action, a read, or a single-node assertion fails when its query finds
|
|
136
|
+
more than one node. Use `.filter()`, `.first()`, or `.nth()` to find one
|
|
137
|
+
node.
|
|
138
|
+
- A direct read, such as `textContent()`, does not retry.
|
|
139
|
+
- Put an `expect` after an `agent.act` goal. It examines the exact result.
|
|
140
|
+
The [replay cache](/cache) records a goal only after a later check passes.
|
|
141
|
+
|
|
142
|
+
See the [screen reference](/reference/screen) and the
|
|
143
|
+
[expect reference](/reference/expect#locator-matchers). For routes, cookies,
|
|
144
|
+
dialogs, and downloads, see [Working with the browser](/browser).
|
|
145
|
+
|
|
146
|
+
## Structure
|
|
147
|
+
|
|
148
|
+
```ts
|
|
149
|
+
import { beforeEach, describe, test } from 'e2e';
|
|
150
|
+
|
|
151
|
+
describe('billing', { tags: ['billing'] }, () => {
|
|
152
|
+
beforeEach(async ({ app }) => { await app.open('/billing'); });
|
|
153
|
+
|
|
154
|
+
test('upgrades', { retries: 2, timeout: 60_000 }, async ({ agent }) => {
|
|
155
|
+
await agent.act('upgrade to Pro');
|
|
156
|
+
});
|
|
157
|
+
});
|
|
158
|
+
|
|
159
|
+
test('needs two organizations', async ({ app }) => {
|
|
160
|
+
test.skip((await countOrganizations()) < 2, 'nothing to switch to');
|
|
161
|
+
await app.open('/organizations');
|
|
162
|
+
});
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
- Each test runs in its own attempt. Prepare its data in fixtures or hooks.
|
|
166
|
+
- `test.skip(condition, reason)` stops the test when the condition is true.
|
|
167
|
+
The report shows the test as `skipped`.
|
|
168
|
+
- For a flow across many tests, set `{ serial: true }` on the group. The
|
|
169
|
+
tests run in sequence on one worker, and retry together.
|
|
170
|
+
- For setup and cleanup, use `test.extend`. See
|
|
171
|
+
[Your own fixtures](/reference/test#your-own-fixtures).
|
|
172
|
+
- Import `test` from `@e2e-dev/web` or `@e2e-dev/mobile` to have the
|
|
173
|
+
`browser` or `device` fixture typed.
|
|
174
|
+
|
|
175
|
+
The [test reference](/reference/test) shows all options.
|
|
176
|
+
|
|
177
|
+
## Run one test
|
|
178
|
+
|
|
179
|
+
<CodeGroup>
|
|
180
|
+
```bash npm
|
|
181
|
+
npx e2e run tests/signup.e2e.ts:12 # the test at line 12
|
|
182
|
+
npx e2e run --grep checkout # tests with a matching title
|
|
183
|
+
npx e2e run --target web # one target
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
```bash pnpm
|
|
187
|
+
pnpm exec e2e run tests/signup.e2e.ts:12 # the test at line 12
|
|
188
|
+
pnpm exec e2e run --grep checkout # tests with a matching title
|
|
189
|
+
pnpm exec e2e run --target web # one target
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
```bash bun
|
|
193
|
+
bunx e2e run tests/signup.e2e.ts:12 # the test at line 12
|
|
194
|
+
bunx e2e run --grep checkout # tests with a matching title
|
|
195
|
+
bunx e2e run --target web # one target
|
|
196
|
+
```
|
|
197
|
+
</CodeGroup>
|
|
198
|
+
|
|
199
|
+
The [CLI reference](/reference/cli#e2e-run) shows all flags.
|
|
200
|
+
|
|
201
|
+
<CardGroup cols={2}>
|
|
202
|
+
<Card title="Caching" icon="box-archive" href="/cache">
|
|
203
|
+
Replay passed goals with no model calls.
|
|
204
|
+
</Card>
|
|
205
|
+
<Card title="How agent steps work" icon="list-check" href="/agent-steps">
|
|
206
|
+
Verdicts, exit codes, and what the model sees.
|
|
207
|
+
</Card>
|
|
208
|
+
</CardGroup>
|
package/docs/debugging.mdx
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Debugging a run
|
|
3
3
|
description: Read the report, open the evidence, watch the browser, and see what the model was sent.
|
|
4
|
+
icon: "magnifying-glass"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
Start with the failure in the terminal. Every run also writes
|
|
@@ -63,7 +64,9 @@ URL: `http://127.0.0.1:3000/todos`
|
|
|
63
64
|
|
|
64
65
|
`Look at` points to the failing call. Assertion failures include expected
|
|
65
66
|
and observed values; locator failures include the requested locator and
|
|
66
|
-
nearby matches. Repeated failure details help you compare attempts.
|
|
67
|
+
nearby matches. Repeated failure details help you compare attempts. The
|
|
68
|
+
runner spends at most 5 s capturing the screen and screenshot once per failed
|
|
69
|
+
attempt; an app that does not answer in time gets no screen section.
|
|
67
70
|
|
|
68
71
|
## The report
|
|
69
72
|
|
|
@@ -105,9 +108,9 @@ reruns names, and writes its own under `rerun-<n>/`:
|
|
|
105
108
|
`--trace off` skips the cost for one run, `--trace` records every attempt
|
|
106
109
|
of a CI run.
|
|
107
110
|
- **Downloads** saved through `browser.waitForDownload`, recorded with kind
|
|
108
|
-
`download`. They are the app's bytes: `redaction: "incomplete"` unless
|
|
109
|
-
secret
|
|
110
|
-
|
|
111
|
+
`download`. They are the app's bytes: `redaction: "incomplete"` unless the
|
|
112
|
+
session knows a secret value and the file is text, which the runner rewrites and
|
|
113
|
+
marks `complete`.
|
|
111
114
|
- **Videos** and **agent transcripts** when you ask for them; see below.
|
|
112
115
|
|
|
113
116
|
After a secret fill, no screenshots are taken for the rest of the attempt; see
|
|
@@ -196,9 +199,16 @@ stdout | web tests/todos.e2e.ts > adds a todo
|
|
|
196
199
|
hello from the test
|
|
197
200
|
```
|
|
198
201
|
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
+
Output from outside a test, such as a test file's top level or a reporter,
|
|
203
|
+
prints under `stdout | runner` or `stderr | runner`. Both streams print on
|
|
204
|
+
stdout, in a pipe or a CI log too. Without the `list` reporter, output
|
|
205
|
+
goes to its own stream as written.
|
|
206
|
+
|
|
207
|
+
Secret values are [redacted](/security#redaction). Output printed while the
|
|
208
|
+
config loads appears once it has loaded, before the run header, and is
|
|
209
|
+
dropped for a warning if the load fails. Every other line appears when it
|
|
210
|
+
is written. The output is not part of the report; use `expect` or an
|
|
211
|
+
artifact for anything a later reader needs.
|
|
202
212
|
|
|
203
213
|
## Agent steps
|
|
204
214
|
|
|
@@ -264,13 +274,13 @@ large raw JSON file.
|
|
|
264
274
|
|
|
265
275
|
Replayed steps make no model call and leave no AI trace. Add `--no-cache`
|
|
266
276
|
when you need to inspect the whole flow or rule out a stale recording. See
|
|
267
|
-
[Caching
|
|
277
|
+
[Caching](/cache).
|
|
268
278
|
|
|
269
279
|
## Common errors
|
|
270
280
|
|
|
271
281
|
| Code | Usual cause | Fix |
|
|
272
282
|
| --- | --- | --- |
|
|
273
|
-
| `APP_UNREACHABLE` | Nothing answers at `app.url`, or `app.command` never became ready; on a device, the message may name the iOS automation runner instead (busy, wedged, or a snapshot it could not present) | Set `app.command.log` and read it; check the port; raise `app.command.startupTimeout`. For the runner, follow the recovery in the message: `npx agent-device daemon stop`, reboot the simulator (see [
|
|
283
|
+
| `APP_UNREACHABLE` | Nothing answers at `app.url`, or `app.command` never became ready; on a device, the message may name the iOS automation runner instead (busy, wedged, or a snapshot it could not present) | Set `app.command.log` and read it; check the port; raise `app.command.startupTimeout`. For the runner, follow the recovery in the message: `npx agent-device daemon stop`, reboot the simulator (see [Mobile](/mobile#troubleshooting)) |
|
|
274
284
|
| `APP_ALREADY_RUNNING` | Something already serves `app.url` when the runner wanted to start `app.command` | Stop it, or set `reuseExisting: true` for local runs |
|
|
275
285
|
| `LOCATOR_NOT_FOUND` | Wrong role or name, text not exact, element inside an iframe, page not open | Read the markup for the accessible name; try `exact: false`; `browser.frameLocator` for iframes; `--headed` to look |
|
|
276
286
|
| `LOCATOR_AMBIGUOUS` | Two matches: a hidden duplicate, a repeated label | Add `{ name }`, scope under a container, `filter`, `first()`, or `{ visible: true }` |
|
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Decision models
|
|
3
|
+
description: Run bounded actions with a decision model and a small text model instead of a full LLM agent.
|
|
4
|
+
icon: "scale-balanced"
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
`@e2e-dev/decision` runs `agent.act` and `agent.assert` through a decision
|
|
8
|
+
model instead of an LLM. For each action the decision model answers one
|
|
9
|
+
request: which operation, and which element for each operation that has
|
|
10
|
+
more than one, chosen from what the executor builds out of the latest
|
|
11
|
+
semantic tree. It never generates free text: when the choice is `type`, a
|
|
12
|
+
small language model writes the field value. Both models are
|
|
13
|
+
plain AI SDK models, so any decision model provider that answers `choice`
|
|
14
|
+
questions with probability distributions (Jev) and any language-model
|
|
15
|
+
provider works; the runner still authorizes every dispatched action.
|
|
16
|
+
|
|
17
|
+
## Install and configure
|
|
18
|
+
|
|
19
|
+
<CodeGroup>
|
|
20
|
+
```bash npm
|
|
21
|
+
npm install -D @e2e-dev/decision
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
```bash pnpm
|
|
25
|
+
pnpm add -D @e2e-dev/decision
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
```bash bun
|
|
29
|
+
bun add -d @e2e-dev/decision
|
|
30
|
+
```
|
|
31
|
+
</CodeGroup>
|
|
32
|
+
|
|
33
|
+
The decision model answers `choice` questions with probability
|
|
34
|
+
distributions; the text model writes field values. Each installs as its
|
|
35
|
+
own package. Decision models need `ai` 7, version 7.0.128 or later:
|
|
36
|
+
|
|
37
|
+
<CodeGroup>
|
|
38
|
+
```bash npm
|
|
39
|
+
npm install -D @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
```bash pnpm
|
|
43
|
+
pnpm add -D @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
```bash bun
|
|
47
|
+
bun add -d @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider
|
|
48
|
+
```
|
|
49
|
+
</CodeGroup>
|
|
50
|
+
|
|
51
|
+
```ts title="e2e.config.ts"
|
|
52
|
+
import type { E2EConfig } from 'e2e';
|
|
53
|
+
import { web } from '@e2e-dev/web';
|
|
54
|
+
import { decisionExecutor } from '@e2e-dev/decision';
|
|
55
|
+
import { typeSafeAi } from '@ai-sdk/typesafe-ai';
|
|
56
|
+
import { openrouter } from '@openrouter/ai-sdk-provider';
|
|
57
|
+
|
|
58
|
+
export default {
|
|
59
|
+
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
60
|
+
agents: {
|
|
61
|
+
default: {
|
|
62
|
+
executor: decisionExecutor({
|
|
63
|
+
model: typeSafeAi.decisionModel('jev-latest'),
|
|
64
|
+
textModel: openrouter('inception/mercury-2.5'),
|
|
65
|
+
}),
|
|
66
|
+
},
|
|
67
|
+
},
|
|
68
|
+
} satisfies E2EConfig;
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
`evaluationModel()` instances still work. A model id string or a language
|
|
72
|
+
model in `model` fails config load with `INVALID_CONFIG`.
|
|
73
|
+
|
|
74
|
+
Tests stay plain natural language, with no params needed:
|
|
75
|
+
|
|
76
|
+
```ts title="tests/todos.e2e.ts"
|
|
77
|
+
import { test, expect } from 'e2e';
|
|
78
|
+
|
|
79
|
+
test('adds a todo', async ({ app, screen, agent }) => {
|
|
80
|
+
await app.open('/todos');
|
|
81
|
+
await agent.act('Add a todo named Buy coffee, and verify it is listed.');
|
|
82
|
+
await expect(screen.getByTestId('todo')).toContainText('Buy coffee');
|
|
83
|
+
await agent.assert('The todo list contains Buy coffee.');
|
|
84
|
+
});
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## How text gets typed
|
|
88
|
+
|
|
89
|
+
The decision model only ever names an element index. When it picks `type`,
|
|
90
|
+
the text model writes the value from the goal, the field's label, and the
|
|
91
|
+
page text, and the executor types exactly that string. Model output never
|
|
92
|
+
becomes a selector, a URL, or a key. When the goal supplies no value,
|
|
93
|
+
nothing is typed and the decision model picks another action. Password
|
|
94
|
+
fields never reach the text model; secrets stay declared handles that only
|
|
95
|
+
`typeSecret` fills.
|
|
96
|
+
|
|
97
|
+
The text model is also the agent's judgment tier: it is exposed as the
|
|
98
|
+
executor's `model`, so `waitFor` and `extract` use it too. It can come
|
|
99
|
+
from the agents entry's `model` instead. Without either, `type` is never
|
|
100
|
+
offered.
|
|
101
|
+
|
|
102
|
+
## Finishing a step
|
|
103
|
+
|
|
104
|
+
When the decision model answers `done` or `failed`, the executor checks
|
|
105
|
+
the claim on a fresh screen with one question: is the step's task complete?
|
|
106
|
+
It sees the step's non-secret params, the actions taken with the non-secret
|
|
107
|
+
values typed, and the page, including the text of named status regions, but
|
|
108
|
+
no model reasoning or secret. A confirmed `done` passes the
|
|
109
|
+
step and a confirmed `failed` fails it with `ACTION_FAILED`. An
|
|
110
|
+
unconfirmed claim goes back to the decision model as history; the second one
|
|
111
|
+
ends the step, as `ACTION_FAILED` when the screen contradicts the goal and
|
|
112
|
+
as `blocked` (`AUTOMATION_UNSUPPORTED`) otherwise.
|
|
113
|
+
|
|
114
|
+
## Gates
|
|
115
|
+
|
|
116
|
+
Both gates are off by default: the executor follows the most probable
|
|
117
|
+
choice. Set `minProbability` or `minConfidence` to block weaker actions,
|
|
118
|
+
reject weaker completion checks, and fail weaker assertions as
|
|
119
|
+
inconclusive instead:
|
|
120
|
+
|
|
121
|
+
```ts title="Executor setup"
|
|
122
|
+
const executor = decisionExecutor({ model, minProbability: 0.9, minConfidence: 0.8 });
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Probability and confidence are different provider fields: probability is the
|
|
126
|
+
selected option's share of the answer distribution, confidence is the
|
|
127
|
+
provider's separate statistic. A threshold is a policy choice; measure
|
|
128
|
+
accuracy on your own tests before changing it.
|
|
129
|
+
|
|
130
|
+
## Limits
|
|
131
|
+
|
|
132
|
+
- No vision: screenshots never reach the model; canvas-only controls and
|
|
133
|
+
image-based targets are not implemented.
|
|
134
|
+
- No `navigate`: in-app navigation is a `tap` on a link. URLs belong in
|
|
135
|
+
test code (`browser.goto`, `app.open`).
|
|
136
|
+
- No multi-select `listbox` in v1.
|
|
137
|
+
- The choice cap is 255 options per question (TypeSafe's limit). Larger
|
|
138
|
+
screens list viewport nodes first and count the rest as omitted, so the
|
|
139
|
+
model can scroll; the step never blocks just for being large.
|
|
140
|
+
- Drag, upload, and hover are not offered in v1.
|
|
141
|
+
- `waitFor` and `extract` use the text model through the runner's
|
|
142
|
+
built-in judgment tier.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Example projects
|
|
3
|
+
description: Run standalone e2e examples with Vite, Next.js, Expo, and SwiftUI.
|
|
4
|
+
icon: "folder-open"
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
Each example is a standalone greeting app with three locator tests and two
|
|
8
|
+
agent tests. Enter a name, press Greet, and check the result. The projects
|
|
9
|
+
install published e2e packages from npm and work outside this repository.
|
|
10
|
+
|
|
11
|
+
<CardGroup cols={2}>
|
|
12
|
+
<Card title="Vite" icon="/images/examples/vite.svg" href="https://github.com/tester-army/e2e/tree/main/examples/with-vite" cta="View example" arrow>
|
|
13
|
+
React with Vite. Run locator and agent tests in Chromium.
|
|
14
|
+
</Card>
|
|
15
|
+
<Card title="Next.js" icon="/images/examples/nextjs.svg" href="https://github.com/tester-army/e2e/tree/main/examples/with-next" cta="View example" arrow>
|
|
16
|
+
Next.js App Router. The runner starts the app and tests it in Chromium.
|
|
17
|
+
</Card>
|
|
18
|
+
<Card title="Expo" icon="/images/integrations/expo.svg" href="https://github.com/tester-army/e2e/tree/main/examples/with-expo" cta="View example" arrow>
|
|
19
|
+
React Native with Expo. Run the same tests on web, an iOS simulator, or an Android emulator.
|
|
20
|
+
</Card>
|
|
21
|
+
<Card title="SwiftUI" icon="/images/examples/swiftui.png" href="https://github.com/tester-army/e2e/tree/main/examples/with-swiftui" cta="View example" arrow>
|
|
22
|
+
A native iOS app with an Xcode project. Run tests on an iPhone simulator.
|
|
23
|
+
</Card>
|
|
24
|
+
</CardGroup>
|
|
25
|
+
|
|
26
|
+
Follow each project's README for installation, builds, and run commands.
|
|
27
|
+
The config and tests live beside the app. The SwiftUI project keeps them
|
|
28
|
+
in its `e2e/` folder.
|
|
29
|
+
|
|
30
|
+
Locator tests need no model. Set `AI_GATEWAY_API_KEY` to run the agent
|
|
31
|
+
tests; without it, those tests skip. See [Models](/models) for model setup
|
|
32
|
+
and the [Quickstart](/quickstart) to add e2e to your own app.
|
package/docs/executors.mdx
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Custom executors
|
|
3
3
|
description: Customize the agent loop or implement your own step executor.
|
|
4
|
+
icon: "microchip"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
Use a custom executor when a `system` prompt and [project tools](/tools) are
|
|
@@ -14,6 +15,8 @@ not enough. There are two options:
|
|
|
14
15
|
`ai` package installed.
|
|
15
16
|
|
|
16
17
|
Both receive a deadline, redacted observations, and checked action methods.
|
|
18
|
+
For a provider that returns bounded choices instead of generated text, see
|
|
19
|
+
the [Clef and Jev decision executor](/decision-models).
|
|
17
20
|
The runner records their verdicts. Either goes under `executor` in an agents
|
|
18
21
|
entry, beside the options that still apply: `model`, `judge`, `context`, and
|
|
19
22
|
the budgets. An entry with `executor` rejects `system` and `tools`; pass
|
package/docs/explore.mdx
CHANGED
package/docs/github.mdx
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Pull request comments
|
|
3
3
|
description: The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.
|
|
4
|
+
icon: "code-pull-request"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
`@e2e-dev/github` posts test results as a pull request comment, including
|
|
@@ -73,7 +74,7 @@ that path. The checkout root is the nearest directory holding `.git` inside
|
|
|
73
74
|
|
|
74
75
|
## The comment
|
|
75
76
|
|
|
76
|
-
```markdown
|
|
77
|
+
```markdown expandable
|
|
77
78
|
### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
|
|
78
79
|
9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31
|
|
79
80
|
|
package/docs/index.mdx
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Introduction
|
|
3
3
|
description: Open source AI testing for web and mobile apps. Spend less time on tests and more time building.
|
|
4
|
+
icon: "house"
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
import { VideoCard } from './snippets/video-card.jsx';
|
|
@@ -9,9 +10,9 @@ e2e is a next-generation testing framework for web and mobile
|
|
|
9
10
|
apps, built to keep up with the speed we ship software today.
|
|
10
11
|
|
|
11
12
|
You choose how agentic each test is: fully deterministic with
|
|
12
|
-
[locators](/locators) and
|
|
13
|
+
[locators](/core-concepts#locators) and
|
|
13
14
|
[assertions](/reference/expect#locator-matchers), fully driven by an agent
|
|
14
|
-
from [goals](/goals) in natural language, or a mix of
|
|
15
|
+
from [goals](/core-concepts#goals) in natural language, or a mix of
|
|
15
16
|
both in the same test.
|
|
16
17
|
|
|
17
18
|
<VideoCard
|
|
@@ -49,7 +50,7 @@ you shipped.
|
|
|
49
50
|
## How it works
|
|
50
51
|
|
|
51
52
|
- **Targets.** Each test runs once per target, such as Chromium or an iPhone
|
|
52
|
-
simulator. The [web engine](/
|
|
53
|
+
simulator. The [web engine](/web) drives browsers through
|
|
53
54
|
Playwright, the [mobile engine](/mobile) drives iOS simulators and Android
|
|
54
55
|
emulators through agent-device, and you can
|
|
55
56
|
[write your own](/writing-an-engine).
|
|
@@ -70,8 +71,11 @@ you shipped.
|
|
|
70
71
|
<Card title="Quickstart" icon="rocket" href="/quickstart">
|
|
71
72
|
One prompt for your coding agent, or set up by hand.
|
|
72
73
|
</Card>
|
|
73
|
-
<Card title="
|
|
74
|
-
|
|
74
|
+
<Card title="Example projects" icon="folder-open" href="/example-projects">
|
|
75
|
+
Run a standalone app with Vite, Next.js, Expo, or SwiftUI.
|
|
76
|
+
</Card>
|
|
77
|
+
<Card title="Core concepts" icon="pen-to-square" href="/core-concepts">
|
|
78
|
+
Goals, assertions, and locators.
|
|
75
79
|
</Card>
|
|
76
80
|
<Card title="Coding agents" icon="robot" href="/coding-agents">
|
|
77
81
|
Let your coding agent write, run, and debug tests.
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
title: EAS Simulators
|
|
3
3
|
sidebarTitle: EAS Simulators
|
|
4
4
|
description: Run mobile targets on EAS Simulators' hosted iOS simulators and Android emulators with the easSimulators() device provider from @e2e-dev/eas.
|
|
5
|
+
icon: "/images/icons/expo.svg"
|
|
5
6
|
---
|
|
6
7
|
|
|
7
8
|
`easSimulators()` from `@e2e-dev/eas` is a
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
title: Kernel browsers
|
|
3
3
|
sidebarTitle: Kernel
|
|
4
4
|
description: Run web targets in Kernel's hosted Chromium with the kernel() browser provider from @e2e-dev/kernel.
|
|
5
|
+
icon: "/images/icons/kernel.svg"
|
|
5
6
|
---
|
|
6
7
|
|
|
7
8
|
`kernel()` from `@e2e-dev/kernel` is a
|
|
@@ -60,7 +61,7 @@ web({
|
|
|
60
61
|
```
|
|
61
62
|
|
|
62
63
|
Each attempt's window then opens maximized, and the page
|
|
63
|
-
[fills it](/
|
|
64
|
+
[fills it](/web#fill-the-window): 1920 by 993 on a 1920 by 1080 screen,
|
|
64
65
|
after the tab strip and address bar. With a fixed web `viewport`, the page is
|
|
65
66
|
emulated at that size inside a smaller window, so Kernel's live view shows it
|
|
66
67
|
cut off or in part of the window.
|
package/docs/migrate/cypress.mdx
CHANGED
|
@@ -1,8 +1,31 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: From Cypress
|
|
3
3
|
description: Replace Cypress command chains with async tests and keep your data-cy attributes.
|
|
4
|
+
icon: "/images/icons/cypress.svg"
|
|
4
5
|
---
|
|
5
6
|
|
|
7
|
+
import { SetupPrompt } from '../snippets/setup-prompt-card.jsx';
|
|
8
|
+
|
|
9
|
+
export const migrationPrompt = `Migrate this project's Cypress tests to e2e, one spec at a time. Docs: \`https://e2e.tester.army/docs/migrate/cypress.md\`
|
|
10
|
+
|
|
11
|
+
1. Read the migration guide above. It maps Cypress config, commands, and assertions to e2e.
|
|
12
|
+
2. If the project has no e2e config (\`e2e.config.ts\` or \`e2e.config.mts\`), run \`npx e2e init --yes\` (or the pnpm or bun equivalent). It writes a config and an example test, and installs nothing yet.
|
|
13
|
+
3. Read the e2e skill (\`npx e2e guide\` prints it, \`npx e2e guide <topic>\` prints a topic) and follow it for every step below.
|
|
14
|
+
4. If you ran init, ask me which model to use for agent steps: a ChatGPT, GitHub Copilot, OpenCode Console, or SuperGrok subscription, an API key, or a local model. Configure it with its provider package. If it needs a sign-in, give me the \`npx e2e login\` command and wait until I confirm.
|
|
15
|
+
5. Port \`cypress.config.*\` into the e2e config: \`baseUrl\` and the dev server command become the target's \`app\`, and if the specs select by \`data-cy\`, set \`web({ testIdAttribute: 'data-cy' })\` so \`screen.getByTestId\` reads it. Keep Cypress installed while both suites run.
|
|
16
|
+
6. Install dependencies.
|
|
17
|
+
7. List the Cypress end-to-end specs and ask me which one to move first. Suggest one that breaks often or waits with \`cy.wait(ms)\`. Component tests have no e2e equivalent; leave them in Cypress.
|
|
18
|
+
8. Rewrite that spec as a \`.e2e.ts\` file under \`tests/\` (e2e runs \`tests/**/*.e2e.ts\` by default), with one awaited call per line and no chains. \`cy.contains\` matches a substring and e2e queries match the whole text, so pass \`{ exact: false }\` where the spec relied on a fragment. Replace each \`cy.wait(ms)\` with an assertion on the state the test waits for. Move \`cy.task\`, \`cy.exec\`, and \`cy.request\` into plain Node code, and custom commands into functions or fixtures.
|
|
19
|
+
9. Keep navigation and the final assertions deterministic, with \`app.open()\` and locator assertions. Where a run of scripted interactions breaks often, propose an \`agent.act\` goal for it instead.
|
|
20
|
+
10. Run the new file with \`npx e2e run <file>\` and fix it until it passes. Keep the Cypress spec until I confirm it can go.
|
|
21
|
+
11. Summarize which tests moved, which stayed in Cypress, and why.`;
|
|
22
|
+
|
|
23
|
+
<SetupPrompt>
|
|
24
|
+
|
|
25
|
+
<Prompt description={migrationPrompt} children={migrationPrompt} actions={["copy", "cursor"]} />
|
|
26
|
+
|
|
27
|
+
</SetupPrompt>
|
|
28
|
+
|
|
6
29
|
e2e tests run in Node. Use `await` for UI actions and call Node functions
|
|
7
30
|
directly for work previously handled by `cy.task`. Both suites can stay in
|
|
8
31
|
the project while you migrate one spec at a time.
|
|
@@ -15,7 +38,7 @@ for the agent calls shown below.
|
|
|
15
38
|
| Cypress | e2e |
|
|
16
39
|
| --- | --- |
|
|
17
40
|
| `baseUrl` | `app: { url }` on the target |
|
|
18
|
-
| `start-server-and-test` | `app: { url, command }`, see [Starting your app](/starting-your-app) |
|
|
41
|
+
| `start-server-and-test` | `app: { url, command }`, see [Starting your app](/web#starting-your-app) |
|
|
19
42
|
| `viewportWidth`, `viewportHeight` | `web({ viewport })` |
|
|
20
43
|
| `data-cy` selectors | `web({ testIdAttribute: 'data-cy' })`, then `screen.getByTestId` |
|
|
21
44
|
| `retries` | `retries` |
|
|
@@ -115,8 +138,8 @@ Start with a spec that often breaks and whose fixed `cy.wait` calls can
|
|
|
115
138
|
become assertions on the condition the test needs.
|
|
116
139
|
|
|
117
140
|
<CardGroup cols={2}>
|
|
118
|
-
<Card title="
|
|
119
|
-
|
|
141
|
+
<Card title="Core concepts" icon="pen-to-square" href="/core-concepts">
|
|
142
|
+
Goals, assertions, and locators instead of scripted selectors.
|
|
120
143
|
</Card>
|
|
121
144
|
<Card title="Working with the browser" icon="globe" href="/browser">
|
|
122
145
|
Routes, dialogs, downloads, iframes, and page evaluation.
|