e2e 0.21.1 → 0.21.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -0
- package/dist/.build.json +1 -1
- package/dist/agent/act-validation.js +1 -0
- package/dist/agent/act.js +7 -3
- package/dist/agent/action-dispatcher.js +30 -8
- package/dist/agent/actions.d.ts +2 -0
- package/dist/agent/actions.js +5 -2
- package/dist/agent/error.js +1 -0
- package/dist/agent/executor.d.ts +14 -6
- package/dist/agent/index.js +5 -1
- package/dist/agent/invocation.d.ts +3 -0
- package/dist/agent/model/sdk.d.ts +5 -0
- package/dist/agent/model/sdk.js +18 -1
- package/dist/agent/observation.js +1 -0
- package/dist/agent/primitives.js +14 -3
- package/dist/agent/replay.js +9 -3
- package/dist/agent/step-cache.d.ts +3 -2
- package/dist/agent/step-cache.js +7 -4
- package/dist/agent/tool-loop.js +4 -1
- package/dist/cache/context.js +1 -1
- package/dist/cache/recorder.d.ts +2 -1
- package/dist/cache/recorder.js +11 -2
- package/dist/cache/trace-fields.d.ts +10 -0
- package/dist/cache/trace-fields.js +17 -0
- package/dist/cache/trace.d.ts +9 -1
- package/dist/cache/trace.js +26 -7
- package/dist/cli/feedback.js +2 -1
- package/dist/cli/init/sibling-versions.json +1 -1
- package/dist/collect/select.js +12 -2
- package/dist/config/agent.d.ts +1 -1
- package/dist/config/diagnose.js +9 -3
- package/dist/config/esm-hooks.js +3 -2
- package/dist/config/resolve.js +3 -2
- package/dist/config/typescript.js +52 -2
- package/dist/engine/contract.d.ts +3 -2
- package/dist/engine/manifest.js +2 -2
- package/dist/internal/errors.js +5 -2
- package/dist/internal/source.js +2 -0
- package/dist/internal/unserved-model.d.ts +5 -0
- package/dist/internal/unserved-model.js +22 -0
- package/dist/locator/screen.js +1 -2
- package/dist/oauth/copilot.js +23 -5
- package/dist/oauth/device-code.js +35 -19
- package/dist/oauth/errors.d.ts +3 -0
- package/dist/oauth/errors.js +26 -11
- package/dist/oauth/providers/github-copilot.d.ts +4 -0
- package/dist/oauth/providers/github-copilot.js +64 -16
- package/dist/oauth/providers/openai.d.ts +2 -2
- package/dist/oauth/providers/openai.js +47 -19
- package/dist/oauth/providers/opencode-console.js +36 -15
- package/dist/oauth/providers/xai.js +18 -9
- package/dist/oauth/token-endpoint.d.ts +15 -6
- package/dist/oauth/token-endpoint.js +18 -6
- package/dist/report/code-frame.js +9 -3
- package/dist/report/failure-text.d.ts +10 -2
- package/dist/report/failure-text.js +24 -8
- package/dist/report/list-model.d.ts +13 -0
- package/dist/report/list-steps.d.ts +6 -1
- package/dist/report/list-steps.js +29 -0
- package/dist/report/list.d.ts +1 -0
- package/dist/report/list.js +16 -7
- package/dist/report/live-window.js +2 -1
- package/dist/report/markdown.js +1 -1
- package/dist/run/fixtures.js +2 -0
- package/dist/run/host-trace.d.ts +3 -2
- package/dist/run/host-trace.js +2 -1
- package/dist/run/last-run.d.ts +9 -0
- package/dist/run/last-run.js +28 -0
- package/dist/run/open-session.d.ts +3 -1
- package/dist/run/open-session.js +1 -1
- package/dist/run/runner.js +6 -2
- package/dist/run/sessions.js +3 -3
- package/dist/run/standalone-attempt.d.ts +2 -1
- package/dist/run/standalone-attempt.js +3 -1
- package/dist/run/steps.d.ts +3 -1
- package/dist/run/units.d.ts +8 -0
- package/dist/run/units.js +14 -0
- package/dist/run/worker-models.d.ts +2 -0
- package/dist/run/worker-models.js +7 -0
- package/dist/types.d.ts +2 -2
- package/docs/authentication.mdx +26 -23
- package/docs/browser.mdx +7 -4
- package/docs/bug-bash.mdx +2 -2
- package/docs/cache.mdx +6 -1
- package/docs/ci/bitrise.mdx +1 -1
- package/docs/ci/codemagic.mdx +2 -2
- package/docs/ci/eas-workflows.mdx +1 -1
- package/docs/coding-agents.mdx +7 -35
- package/docs/debugging.mdx +24 -36
- package/docs/executors.mdx +1 -1
- package/docs/github.mdx +2 -1
- package/docs/integrations/eas.mdx +1 -1
- package/docs/integrations/index.mdx +6 -0
- package/docs/integrations/lightpanda.mdx +192 -0
- package/docs/integrations/testmu.mdx +152 -0
- package/docs/mcp.mdx +123 -0
- package/docs/migrate/detox.mdx +1 -1
- package/docs/migrate/maestro.mdx +101 -105
- package/docs/migrate/playwright.mdx +65 -70
- package/docs/mobile.mdx +18 -3
- package/docs/models.mdx +4 -2
- package/docs/quickstart.mdx +7 -0
- package/docs/reference/agent.mdx +369 -277
- package/docs/reference/app.mdx +141 -32
- package/docs/reference/cli.mdx +674 -558
- package/docs/reference/config.mdx +1406 -752
- package/docs/reference/engine.mdx +1062 -256
- package/docs/reference/environment.mdx +156 -84
- package/docs/reference/errors.mdx +106 -72
- package/docs/reference/expect.mdx +752 -372
- package/docs/reference/mcp.mdx +366 -141
- package/docs/reference/mobile.mdx +844 -403
- package/docs/reference/reporters.mdx +312 -106
- package/docs/reference/runner.mdx +373 -174
- package/docs/reference/screen.mdx +931 -398
- package/docs/reference/security.mdx +106 -40
- package/docs/reference/test.mdx +626 -273
- package/docs/reference/web.mdx +956 -533
- package/docs/security.mdx +6 -5
- package/docs/snippets/model-provider.jsx +763 -0
- package/docs/snippets/setup-prompt-card.jsx +22 -0
- package/docs/snippets/show-more.jsx +22 -0
- package/docs/snippets/video-card.jsx +34 -0
- package/docs/snippets/video.jsx +25 -0
- package/docs/subscriptions.mdx +10 -3
- package/docs/telemetry.mdx +39 -39
- package/package.json +3 -2
- package/schema/report-v1.schema.json +6 -0
- package/skills/e2e/references/agent.md +8 -3
- package/skills/e2e/references/debugging.md +1 -1
- package/skills/e2e/references/mcp.md +1 -1
- package/skills/e2e/references/setup.md +12 -3
- package/skills/e2e/references/writing-tests.md +1 -1
package/docs/authentication.mdx
CHANGED
|
@@ -8,8 +8,9 @@ icon: "key"
|
|
|
8
8
|
|
|
9
9
|
A setup test signs in one time and saves a named session. Other tests
|
|
10
10
|
restore that session and start signed in. Passwords and other secrets are
|
|
11
|
-
handles: the runner types the value and
|
|
12
|
-
reports, with
|
|
11
|
+
handles: the runner types the value and shows `<secret:name>` in its place
|
|
12
|
+
in model input and reports, with
|
|
13
|
+
[some limits](/reference/security#where-redaction-stops).
|
|
13
14
|
|
|
14
15
|
## Sign in once in a setup test
|
|
15
16
|
|
|
@@ -89,6 +90,9 @@ memory and deletes it at cleanup. See [Sessions](/reference/security#sessions).
|
|
|
89
90
|
Declare each account in config. Read the password from the environment:
|
|
90
91
|
|
|
91
92
|
```ts title="e2e.config.ts"
|
|
93
|
+
import type { E2EConfig } from 'e2e';
|
|
94
|
+
import { web } from '@e2e-dev/web';
|
|
95
|
+
|
|
92
96
|
export default {
|
|
93
97
|
targets: [{ engine: web(), app: { url: 'http://127.0.0.1:3000' } }],
|
|
94
98
|
credentials: {
|
|
@@ -128,10 +132,9 @@ export default {
|
|
|
128
132
|
} satisfies E2EConfig;
|
|
129
133
|
```
|
|
130
134
|
|
|
131
|
-
The value
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
types it, use a string. The same applies to a `secrets` entry.
|
|
135
|
+
The value is redacted only from its first fill on. If the app can show the
|
|
136
|
+
value before a test types it, use a string. The same applies to a `secrets`
|
|
137
|
+
entry.
|
|
135
138
|
|
|
136
139
|
</Accordion>
|
|
137
140
|
|
|
@@ -139,8 +142,7 @@ types it, use a string. The same applies to a `secrets` entry.
|
|
|
139
142
|
|
|
140
143
|
`credentials.user(name)` returns the username as a plain string and the
|
|
141
144
|
password as a `Secret` named `<name>.password`. A `Secret` has no `.value`.
|
|
142
|
-
Pass it to a `fill
|
|
143
|
-
option that declares it, such as [`basicAuth`](/web#protected-previews):
|
|
145
|
+
Pass it to a `fill` or to the `params` of an agent step:
|
|
144
146
|
|
|
145
147
|
```ts
|
|
146
148
|
await agent.act('Sign in', {
|
|
@@ -148,21 +150,19 @@ await agent.act('Sign in', {
|
|
|
148
150
|
});
|
|
149
151
|
```
|
|
150
152
|
|
|
151
|
-
The model sees only the secret's name and purpose.
|
|
152
|
-
requires a password field
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
can't read a secure field, so assert that sign-in succeeded instead.
|
|
153
|
+
The model sees only the secret's name and purpose. When the model fills a
|
|
154
|
+
password, the runner requires a password field and types the value. See the
|
|
155
|
+
[secret fill rules](/reference/security#secret-fills). The agent can't read
|
|
156
|
+
a secure field, so assert that sign-in succeeded instead.
|
|
156
157
|
|
|
157
158
|
After a secret fill, the runner takes no screenshots for the rest of the
|
|
158
|
-
attempt, or in the tests that restore the session it saved.
|
|
159
|
-
[screenshots after a secret fill](/reference/security#screenshots-after-a-secret-fill)
|
|
160
|
-
for how to sign in with screenshots kept.
|
|
159
|
+
attempt, or in the tests that restore the session it saved. To keep them, see
|
|
160
|
+
[screenshots after a secret fill](/reference/security#screenshots-after-a-secret-fill).
|
|
161
161
|
|
|
162
162
|
<Warning>
|
|
163
|
-
Redaction matches
|
|
164
|
-
|
|
165
|
-
|
|
163
|
+
Redaction matches strings. A transformed value, such as the last four
|
|
164
|
+
characters of a key, can still appear. Keep secret entry in setup tests, and
|
|
165
|
+
assert the result of the submit. See
|
|
166
166
|
[where redaction stops](/reference/security#where-redaction-stops).
|
|
167
167
|
</Warning>
|
|
168
168
|
|
|
@@ -171,6 +171,9 @@ entry in setup tests, and assert the result of the submit. See
|
|
|
171
171
|
Use `secrets` for API keys, webhook tokens, and other values a test types:
|
|
172
172
|
|
|
173
173
|
```ts title="e2e.config.ts"
|
|
174
|
+
import type { E2EConfig } from 'e2e';
|
|
175
|
+
import { web } from '@e2e-dev/web';
|
|
176
|
+
|
|
174
177
|
export default {
|
|
175
178
|
targets: [{ engine: web(), app: { url: 'http://127.0.0.1:3000' } }],
|
|
176
179
|
secrets: {
|
|
@@ -195,10 +198,10 @@ A `secrets` entry can fill any editable input. An unconfigured name throws
|
|
|
195
198
|
`SECRET_UNAVAILABLE`. `secrets.get()` returns `secrets` entries only, so
|
|
196
199
|
`credentials.admin` and `secrets.admin` can both exist.
|
|
197
200
|
|
|
198
|
-
In `e2e.config.ts`, `secrets.get()` can pass a secret to an engine option
|
|
199
|
-
such as
|
|
200
|
-
|
|
201
|
-
[Secrets in engine options](/reference/security#secrets-in-engine-options).
|
|
201
|
+
In `e2e.config.ts`, `secrets.get()` can pass a secret to an engine option
|
|
202
|
+
that declares it, such as [`basicAuth`](/web#protected-previews). The config
|
|
203
|
+
load checks the name. Screenshots stay on, so the app's pixels aren't masked.
|
|
204
|
+
See [Secrets in engine options](/reference/security#secrets-in-engine-options).
|
|
202
205
|
|
|
203
206
|
## Override a value per run
|
|
204
207
|
|
package/docs/browser.mdx
CHANGED
|
@@ -201,7 +201,9 @@ bunx @e2e-dev/web install webkit --with-deps
|
|
|
201
201
|
```
|
|
202
202
|
</CodeGroup>
|
|
203
203
|
|
|
204
|
-
`connect` attaches to a remote Chromium over CDP instead of launching one
|
|
204
|
+
`connect` attaches to a remote Chromium over CDP instead of launching one,
|
|
205
|
+
or to another browser that speaks CDP, such as a running
|
|
206
|
+
[Lightpanda](/integrations/lightpanda).
|
|
205
207
|
To set sizes, see [Viewports](/web#viewports).
|
|
206
208
|
|
|
207
209
|
## Hosted browsers
|
|
@@ -215,14 +217,15 @@ when the run no longer uses it.
|
|
|
215
217
|
| --- | --- |
|
|
216
218
|
| [Kernel](/integrations/kernel) | `kernel()` from `@e2e-dev/kernel` |
|
|
217
219
|
| [smol machines](/integrations/smol) microVMs on your own computer | `smol()` from `@e2e-dev/smol` |
|
|
220
|
+
| [Lightpanda](/integrations/lightpanda), the open-source headless browser, on your own machine | `lightpanda()` from `@e2e-dev/lightpanda` |
|
|
218
221
|
| Any other service | Your own provider, a few dozen lines in your project |
|
|
219
222
|
|
|
220
223
|
When the service shows the window in a live view, add `viewport: null` so
|
|
221
224
|
the page [fills the window](/web#fill-the-window).
|
|
222
225
|
|
|
223
|
-
A provider implies
|
|
224
|
-
can't be combined with `connect`. A
|
|
225
|
-
nothing.
|
|
226
|
+
A provider implies a browser the engine drives as Chromium over CDP,
|
|
227
|
+
Chromium itself or Lightpanda, and can't be combined with `connect`. A
|
|
228
|
+
target with nothing to run leases nothing.
|
|
226
229
|
|
|
227
230
|
### Write a provider
|
|
228
231
|
|
package/docs/bug-bash.mdx
CHANGED
|
@@ -78,7 +78,7 @@ Bug bash the checkout and account settings on this branch.
|
|
|
78
78
|
environment, the intended design, or the seed data.
|
|
79
79
|
6. Writes a repro test that asserts the expected behavior for each
|
|
80
80
|
remaining finding. It can start one subagent per area, each with its own
|
|
81
|
-
[`e2e mcp` session](/
|
|
81
|
+
[`e2e mcp` session](/mcp) to check locators.
|
|
82
82
|
7. Runs each repro test. The finding is a bug only when the test fails with
|
|
83
83
|
`ASSERTION_FAILED`.
|
|
84
84
|
8. Reports, in this order: confirmed bugs, production risks that it cannot
|
|
@@ -124,7 +124,7 @@ regression test.
|
|
|
124
124
|
<Card title="Exploring without a test" icon="compass" href="/explore">
|
|
125
125
|
What one `e2e explore` run does, its budgets, and its report.
|
|
126
126
|
</Card>
|
|
127
|
-
<Card title="MCP server" icon="plug" href="/
|
|
127
|
+
<Card title="MCP server" icon="plug" href="/mcp">
|
|
128
128
|
Live sessions for coding agents, several at once.
|
|
129
129
|
</Card>
|
|
130
130
|
</CardGroup>
|
package/docs/cache.mdx
CHANGED
|
@@ -62,7 +62,8 @@ a run leaves the cache unchanged.
|
|
|
62
62
|
An attempt whose only failures are ones where no model answered evicts
|
|
63
63
|
nothing, since these say nothing about the app:
|
|
64
64
|
|
|
65
|
-
- `MODEL_UNAVAILABLE`: no model is configured
|
|
65
|
+
- `MODEL_UNAVAILABLE`: no model is configured, or GitHub Copilot doesn't
|
|
66
|
+
serve it to the account.
|
|
66
67
|
- `MODEL_PROVIDER_FAILED`: the provider returned an error, such as a
|
|
67
68
|
missing or rejected key, a rate limit, or a 5xx, or a request got no
|
|
68
69
|
response within 120 seconds.
|
|
@@ -93,6 +94,10 @@ agent. Each of these changes causes a miss:
|
|
|
93
94
|
Changing the model doesn't cause a miss, and neither does upgrading the
|
|
94
95
|
engine: a node the new version resolves differently misses at replay.
|
|
95
96
|
|
|
97
|
+
Use `--no-cache` to evaluate a new model. Replays expose
|
|
98
|
+
`step.cache.recordedBy` when the entry has provenance; older or unattributed
|
|
99
|
+
entries omit it.
|
|
100
|
+
|
|
96
101
|
A timestamp or a fresh email address would miss on every run. Wrap such
|
|
97
102
|
values in `unique()`, and the runner puts in the current value at replay:
|
|
98
103
|
|
package/docs/ci/bitrise.mdx
CHANGED
package/docs/ci/codemagic.mdx
CHANGED
|
@@ -40,7 +40,7 @@ workflows:
|
|
|
40
40
|
environment:
|
|
41
41
|
groups:
|
|
42
42
|
- e2e
|
|
43
|
-
node:
|
|
43
|
+
node: 24
|
|
44
44
|
xcode: latest
|
|
45
45
|
triggering:
|
|
46
46
|
events:
|
|
@@ -77,7 +77,7 @@ workflows:
|
|
|
77
77
|
environment:
|
|
78
78
|
groups:
|
|
79
79
|
- e2e
|
|
80
|
-
node:
|
|
80
|
+
node: 24
|
|
81
81
|
triggering:
|
|
82
82
|
events:
|
|
83
83
|
- push
|
package/docs/coding-agents.mdx
CHANGED
|
@@ -87,39 +87,10 @@ codex plugin marketplace upgrade e2e
|
|
|
87
87
|
|
|
88
88
|
## Let the agent use the app
|
|
89
89
|
|
|
90
|
-
`e2e mcp`
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
`init` registers the server in `.mcp.json` for Claude Code and
|
|
96
|
-
`.cursor/mcp.json` for Cursor. To register it with Claude Code by hand:
|
|
97
|
-
|
|
98
|
-
<CodeGroup>
|
|
99
|
-
```bash npm
|
|
100
|
-
claude mcp add --scope project e2e -- npx e2e mcp
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
```bash pnpm
|
|
104
|
-
claude mcp add --scope project e2e -- pnpm exec e2e mcp
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
```bash bun
|
|
108
|
-
claude mcp add --scope project e2e -- bunx e2e mcp
|
|
109
|
-
```
|
|
110
|
-
</CodeGroup>
|
|
111
|
-
|
|
112
|
-
For Codex, add the server to the project's `.codex/config.toml`, trust the
|
|
113
|
-
project, and start a new session:
|
|
114
|
-
|
|
115
|
-
```toml
|
|
116
|
-
[mcp_servers.e2e]
|
|
117
|
-
command = "npx"
|
|
118
|
-
args = ["e2e", "mcp"]
|
|
119
|
-
```
|
|
120
|
-
|
|
121
|
-
The server does not run tests. See the [MCP server reference](/reference/mcp)
|
|
122
|
-
for its tools, session rules, and flags.
|
|
90
|
+
`e2e mcp` gives the agent a live session on your app. It checks each
|
|
91
|
+
locator against the screen before it writes the test, and checks its own UI
|
|
92
|
+
changes. `init` registers the server. See [MCP server](/mcp) for setup and
|
|
93
|
+
prompts.
|
|
123
94
|
|
|
124
95
|
## How an agent writes and runs a test
|
|
125
96
|
|
|
@@ -251,8 +222,9 @@ bunx skills add tester-army/e2e --skill migrate-to-e2e
|
|
|
251
222
|
|
|
252
223
|
Then ask the agent for `/migrate-to-e2e`. It ports the slice you pick and
|
|
253
224
|
runs each ported test three times on every target. A `MIGRATION.md` file
|
|
254
|
-
records progress for the next session.
|
|
255
|
-
|
|
225
|
+
records progress for the next session. Optionally, the agent plants a temporary bug
|
|
226
|
+
in the app, such as a wrong checkout total, and checks whether both
|
|
227
|
+
suites fail on it. The agent deletes an old test only when you confirm. It first reads the migration guide for your runner:
|
|
256
228
|
[Playwright](/migrate/playwright), [Cypress](/migrate/cypress),
|
|
257
229
|
[Selenium](/migrate/selenium), [Detox](/migrate/detox), or
|
|
258
230
|
[Maestro](/migrate/maestro).
|
package/docs/debugging.mdx
CHANGED
|
@@ -6,9 +6,8 @@ icon: "magnifying-glass"
|
|
|
6
6
|
"twitter:image": "https://e2e.tester.army/docs/images/og/debugging.png?v=38a51086e1"
|
|
7
7
|
---
|
|
8
8
|
|
|
9
|
-
A failed test
|
|
10
|
-
|
|
11
|
-
summary of the whole run, add the `markdown` reporter. It writes
|
|
9
|
+
A failed test prints its error and the path to its [trace page](#traces). For
|
|
10
|
+
a summary of the whole run, add the `markdown` reporter. It writes
|
|
12
11
|
`.e2e/summary.md`.
|
|
13
12
|
|
|
14
13
|
## Read the failure
|
|
@@ -148,8 +147,8 @@ attempts under each test's `rerun-<n>/`.
|
|
|
148
147
|
| Artifact | When |
|
|
149
148
|
| --- | --- |
|
|
150
149
|
| Screenshots | A failed attempt captures one when the engine can and pixels are allowed. The built-in `agent.assert` captures one after either verdict, unless `screenshot: false`. `app.screenshot('label')` captures one on request. After a secret fill, the runner takes [none](/reference/security#screenshots-after-a-secret-fill) for the rest of the attempt. |
|
|
151
|
-
| Downloads | Files saved through `browser.waitForDownload`, kind `download
|
|
152
|
-
| Videos | When you [record them](#watch-it-happen). Videos mask nothing
|
|
150
|
+
| Downloads | Files saved through `browser.waitForDownload`, kind `download`, as the app served them. [Downloads](/reference/security#downloads) says when the runner masks secrets in them. |
|
|
151
|
+
| Videos | When you [record them](#watch-it-happen). Videos mask nothing. See [where redaction stops](/reference/security#where-redaction-stops). |
|
|
153
152
|
| Agent transcripts | With [`--debug`](#agent-steps). |
|
|
154
153
|
|
|
155
154
|
## For scripts: report.json
|
|
@@ -176,6 +175,14 @@ result holds:
|
|
|
176
175
|
| `appLog` | What the app logged. Each line's `step` is the `index` of its step, absent before the first step. |
|
|
177
176
|
| `artifacts` | The captured files |
|
|
178
177
|
|
|
178
|
+
A serial group's tests have no `attempts` of their own. Their results are in
|
|
179
|
+
`run.serialGroups`, one member for each test in each attempt:
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
jq '.run.serialGroups[] | .attempts[-1].members[]? | select(.status != "passed")
|
|
183
|
+
| {testId, status, error, failure}' .e2e/report.json
|
|
184
|
+
```
|
|
185
|
+
|
|
179
186
|
## Watch it happen
|
|
180
187
|
|
|
181
188
|
<CodeGroup>
|
|
@@ -199,8 +206,7 @@ bunx e2e run --video=retain-on-failure # record every attempt, keep the fail
|
|
|
199
206
|
</CodeGroup>
|
|
200
207
|
|
|
201
208
|
`--video` takes an optional mode. Put test files before it, or write
|
|
202
|
-
`--video=<mode>`.
|
|
203
|
-
whose engine can record, and one notice names the others.
|
|
209
|
+
`--video=<mode>`.
|
|
204
210
|
|
|
205
211
|
Each recording goes into its attempt's directory: WebM per page on web, MP4
|
|
206
212
|
on a device. The failure recap prints the path, or the URL when a hosted
|
|
@@ -211,32 +217,14 @@ In CI, set `video: 'on-first-retry'` in the config. A test that passes the
|
|
|
211
217
|
first time records nothing, and a flaky test leaves a recording of its
|
|
212
218
|
retry.
|
|
213
219
|
|
|
214
|
-
<Accordion title="Set video in the config, on a target, or on a test">
|
|
215
|
-
|
|
216
|
-
```ts
|
|
217
|
-
export default {
|
|
218
|
-
video: 'on-first-retry',
|
|
219
|
-
retries: 1,
|
|
220
|
-
targets: [
|
|
221
|
-
{ name: 'chromium', engine: web(), app: { url } },
|
|
222
|
-
{ name: 'firefox', engine: web({ browser: 'firefox' }), app: { url }, video: 'retain-on-failure' },
|
|
223
|
-
],
|
|
224
|
-
} satisfies E2EConfig;
|
|
225
|
-
```
|
|
226
|
-
|
|
227
220
|
A target's `video` wins over the config's, and `--video` wins over both. A
|
|
228
|
-
test's own option
|
|
229
|
-
|
|
230
|
-
```ts
|
|
231
|
-
test('checkout', { video: 'on' }, async ({ agent }) => {
|
|
232
|
-
await agent.act('Buy the first product in the catalog');
|
|
233
|
-
});
|
|
234
|
-
```
|
|
221
|
+
test's own option, such as `test('checkout', { video: 'on' }, ...)`, wins
|
|
222
|
+
over all of them.
|
|
235
223
|
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
224
|
+
The config's `video` and `--video` apply only to targets whose engine can
|
|
225
|
+
record, and one notice names the others. When a target or a test asks for a
|
|
226
|
+
recording its engine can't make, the run fails with `UNSUPPORTED_ARTIFACT`
|
|
227
|
+
before any test starts.
|
|
240
228
|
|
|
241
229
|
[Recording](/reference/config#recording) has every mode and file.
|
|
242
230
|
|
|
@@ -246,7 +234,7 @@ it, the run fails with `UNSUPPORTED_ARTIFACT` before any test starts.
|
|
|
246
234
|
heading that names the stream, target, file, and test:
|
|
247
235
|
|
|
248
236
|
```text
|
|
249
|
-
stdout | web tests/todos.e2e.ts > adds a todo
|
|
237
|
+
stdout | |web| tests/todos.e2e.ts > adds a todo
|
|
250
238
|
hello from the test
|
|
251
239
|
```
|
|
252
240
|
|
|
@@ -304,17 +292,17 @@ prompts, tool calls, and token usage:
|
|
|
304
292
|
<CodeGroup>
|
|
305
293
|
```bash npm
|
|
306
294
|
npx unbox-ai runs .e2e/ai-trace.json
|
|
307
|
-
npx unbox-ai summary .e2e/ai-trace.json --run
|
|
295
|
+
npx unbox-ai summary .e2e/ai-trace.json --run 0
|
|
308
296
|
```
|
|
309
297
|
|
|
310
298
|
```bash pnpm
|
|
311
299
|
pnpm dlx unbox-ai runs .e2e/ai-trace.json
|
|
312
|
-
pnpm dlx unbox-ai summary .e2e/ai-trace.json --run
|
|
300
|
+
pnpm dlx unbox-ai summary .e2e/ai-trace.json --run 0
|
|
313
301
|
```
|
|
314
302
|
|
|
315
303
|
```bash bun
|
|
316
304
|
bunx unbox-ai runs .e2e/ai-trace.json
|
|
317
|
-
bunx unbox-ai summary .e2e/ai-trace.json --run
|
|
305
|
+
bunx unbox-ai summary .e2e/ai-trace.json --run 0
|
|
318
306
|
```
|
|
319
307
|
</CodeGroup>
|
|
320
308
|
|
|
@@ -333,7 +321,7 @@ to trace the whole flow or rule out a stale [recording](/cache).
|
|
|
333
321
|
| `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about something the screen doesn't show: on another page, still loading, or only in pixels | Open or wait for the right screen first; add `vision: true` when the answer is in pixels (the message says so) |
|
|
334
322
|
| `ACTION_FAILED` | Element covered, disabled, or detached, or an operation timed out | Wait on the right condition with `expect` first; close overlays |
|
|
335
323
|
| `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT` | The agent ran out of actions, model calls, or time | Scope the goal to one step. Raise the agent entry's `maxSteps` or `maxModelCalls` for the budget, the config `timeout` for an `act` step, or the agent's `judgmentTimeout` for `assert`, `waitFor`, and `extract` |
|
|
336
|
-
| `MODEL_UNAVAILABLE` | No model constructed in the config | See [Models](/models) |
|
|
324
|
+
| `MODEL_UNAVAILABLE` | No model constructed in the config, or GitHub Copilot doesn't serve the configured model to your account | See [Models](/models); for Copilot, [Pick a model](/subscriptions#pick-a-model) |
|
|
337
325
|
| `MODEL_PROVIDER_FAILED` | Bad key, rate limit, no credits, or every try of one request went 120 s without an answer | Check the key the provider reads and the quota; for a slow local model, a smaller screen (`agents.<name>.maxObservationBytes`) or a faster model |
|
|
338
326
|
|
|
339
327
|
The [errors reference](/reference/errors) lists every code, its class, and
|
package/docs/executors.mdx
CHANGED
|
@@ -124,7 +124,7 @@ export default {
|
|
|
124
124
|
| `ctx.step` | The `kind`, instruction, params, and `index` in the attempt's timeline. A configured secret's value in the instruction or a param reads `<secret:name>`. A `secrets.get()` handle arrives as a name-only placeholder that `typeSecret` fills |
|
|
125
125
|
| `ctx.observe({ tree, pixels })` | Redacted screen text, with an optional node tree and masked screenshot |
|
|
126
126
|
| `ctx.actions` | Checked and recorded actions: `tap`, `doubleTap`, `longPress`, `secondaryTap`, `hover`, `type`, `typeSecret`, `press`, `select`, `check`, `drag`, `scrollTo`, `scrollUntil`, `upload`, `scroll`, `navigate`, `back`, and the point and keyboard actions below |
|
|
127
|
-
| `ctx.target` | The target's `name`, `platform`, and `verbs`, the actions its engine supports. Offer only these to the model |
|
|
127
|
+
| `ctx.target` | The target's `name`, `platform`, and `verbs`, the actions its engine supports. Offer only these to the model. `tapModifiers` is true when `tap`, `doubleTap`, and `secondaryTap` take `{ modifiers }` |
|
|
128
128
|
| `ctx.pixelsTainted` | True after a [secret fill](/reference/security#screenshots-after-a-secret-fill). Then no screenshots are available |
|
|
129
129
|
| `ctx.signal` | Aborts on cancellation, at the step deadline, or on any other hard stop. The deadline can't stop work that ignores it |
|
|
130
130
|
| `ctx.replayedPrefix` | Set when a cache replay ran part of the step first. See [Replay cache](#replay-cache) |
|
package/docs/github.mdx
CHANGED
|
@@ -182,6 +182,7 @@ post, one row under the summary gives the reason:
|
|
|
182
182
|
| A `pull_request` event from a fork | The token is read-only, so posting fails. The row says why. |
|
|
183
183
|
| An `issue_comment` event | On a pull request's thread, the comment has no source links, because the checked-out commit is on the default branch. |
|
|
184
184
|
| Missing `pull-requests: write` | Posting fails, and the reporter names the permission. |
|
|
185
|
+
| GitHub API rate limit | Posting stops and reports throttling. Wait for the reset or `Retry-After` before rerunning the job. |
|
|
185
186
|
|
|
186
187
|
The reporter reads the token when the run finishes, and the report doesn't
|
|
187
188
|
include it. On GitHub Enterprise Server, the reporter uses
|
|
@@ -190,7 +191,7 @@ include it. On GitHub Enterprise Server, the reporter uses
|
|
|
190
191
|
## Next
|
|
191
192
|
|
|
192
193
|
<CardGroup cols={2}>
|
|
193
|
-
<Card title="GitHub Actions" icon="github" href="/ci/github-actions">
|
|
194
|
+
<Card title="GitHub Actions" icon="/images/integrations/github-actions.svg" href="/ci/github-actions">
|
|
194
195
|
The full workflow: install, browsers, cache, and artifacts.
|
|
195
196
|
</Card>
|
|
196
197
|
<Card title="Debugging a run" icon="magnifying-glass" href="/debugging">
|
|
@@ -127,7 +127,7 @@ The simulator boots with no apps. Choose one way to put yours on it:
|
|
|
127
127
|
An unknown option, `buildId` with `applicationArchiveUrl`, or a
|
|
128
128
|
`maxIdleTimeMinutes` not below a `maxDurationMinutes` passed with it fails
|
|
129
129
|
the config load
|
|
130
|
-
with `INVALID_CONFIG`.
|
|
130
|
+
with `INVALID_CONFIG`. Both minute limits must be non-negative integers.
|
|
131
131
|
|
|
132
132
|
## Sessions
|
|
133
133
|
|
|
@@ -15,6 +15,9 @@ Only the target's `browser` or `device` in the config changes.
|
|
|
15
15
|
<Card title="Kernel" icon="/images/integrations/kernel.svg" href="/integrations/kernel">
|
|
16
16
|
`@e2e-dev/kernel`: hosted Chromium for web targets, one browser per worker or per test.
|
|
17
17
|
</Card>
|
|
18
|
+
<Card title="Lightpanda" icon="/images/integrations/lightpanda.svg" href="/integrations/lightpanda">
|
|
19
|
+
`@e2e-dev/lightpanda`: the open-source headless browser, started per worker slot on your own machine, no Chromium.
|
|
20
|
+
</Card>
|
|
18
21
|
<Card title="EAS Simulators" icon="/images/integrations/expo.svg" href="/integrations/eas">
|
|
19
22
|
`@e2e-dev/eas`: hosted iOS simulators and Android emulators for mobile targets.
|
|
20
23
|
</Card>
|
|
@@ -24,6 +27,9 @@ Only the target's `browser` or `device` in the config changes.
|
|
|
24
27
|
<Card title="Limrun" icon="/images/integrations/limrun.svg" href="/integrations/limrun">
|
|
25
28
|
`@e2e-dev/limrun`: hosted iOS simulators and Android emulators, with app installation and device cleanup.
|
|
26
29
|
</Card>
|
|
30
|
+
<Card title="TestMu AI" icon="/images/integrations/testmu.svg" href="/integrations/testmu">
|
|
31
|
+
`@e2e-dev/testmu`: hosted Android emulators, iOS simulators, and real devices for mobile targets.
|
|
32
|
+
</Card>
|
|
27
33
|
</CardGroup>
|
|
28
34
|
|
|
29
35
|
## Use a service with no integration
|
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Lightpanda
|
|
3
|
+
sidebarTitle: Lightpanda
|
|
4
|
+
description: Run web targets in Lightpanda, the open-source headless browser, with the lightpanda() browser provider from @e2e-dev/lightpanda.
|
|
5
|
+
icon: "/images/icons/lightpanda.svg"
|
|
6
|
+
"og:image": "https://e2e.tester.army/docs/images/og/integrations/lightpanda.png?v=b97a3ebae4"
|
|
7
|
+
"twitter:image": "https://e2e.tester.army/docs/images/og/integrations/lightpanda.png?v=b97a3ebae4"
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
[Lightpanda](https://lightpanda.io) is an open-source headless browser
|
|
11
|
+
written from scratch, without Chromium. It starts in milliseconds, holds a
|
|
12
|
+
fraction of Chrome's memory, and speaks the Chrome DevTools Protocol.
|
|
13
|
+
`lightpanda()` from `@e2e-dev/lightpanda` is a
|
|
14
|
+
[browser provider](/browser#hosted-browsers) for it: each worker slot gets
|
|
15
|
+
its own `lightpanda serve`, started on a free port when the run starts and
|
|
16
|
+
stopped when it ends. Tests and CI stay the same; the config changes only
|
|
17
|
+
in the target's `browser`.
|
|
18
|
+
|
|
19
|
+
Lightpanda has no layout engine. It builds the DOM, runs JavaScript, and
|
|
20
|
+
exposes the accessibility tree the engine reads, which is what locators,
|
|
21
|
+
assertions, and [agent steps](/agent-steps) use. Anything that depends on
|
|
22
|
+
where a node is painted behaves differently; see
|
|
23
|
+
[What does not work](#what-does-not-work) before moving a suite over.
|
|
24
|
+
|
|
25
|
+
## Setup
|
|
26
|
+
|
|
27
|
+
Install the package:
|
|
28
|
+
|
|
29
|
+
<CodeGroup>
|
|
30
|
+
```bash npm
|
|
31
|
+
npm install --save-dev @e2e-dev/lightpanda
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
```bash pnpm
|
|
35
|
+
pnpm add -D @e2e-dev/lightpanda
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
```bash bun
|
|
39
|
+
bun add -d @e2e-dev/lightpanda
|
|
40
|
+
```
|
|
41
|
+
</CodeGroup>
|
|
42
|
+
|
|
43
|
+
And the Lightpanda nightly binary, which the provider starts:
|
|
44
|
+
|
|
45
|
+
<CodeGroup>
|
|
46
|
+
```bash macOS
|
|
47
|
+
brew install lightpanda-io/browser/lightpanda
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
```bash Linux
|
|
51
|
+
curl -L -o lightpanda https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-x86_64-linux
|
|
52
|
+
install -D -m 755 lightpanda ~/.local/bin/lightpanda
|
|
53
|
+
```
|
|
54
|
+
</CodeGroup>
|
|
55
|
+
|
|
56
|
+
```ts title="e2e.config.ts"
|
|
57
|
+
import type { E2EConfig } from 'e2e';
|
|
58
|
+
import { web } from '@e2e-dev/web';
|
|
59
|
+
import { lightpanda } from '@e2e-dev/lightpanda';
|
|
60
|
+
|
|
61
|
+
export default {
|
|
62
|
+
targets: [
|
|
63
|
+
{
|
|
64
|
+
name: 'lightpanda',
|
|
65
|
+
engine: web({ browser: lightpanda() }),
|
|
66
|
+
app: { url: 'http://localhost:3000' },
|
|
67
|
+
},
|
|
68
|
+
],
|
|
69
|
+
} satisfies E2EConfig;
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
The provider runs `lightpanda` from `PATH`, `~/.lightpanda/lightpanda`, or
|
|
73
|
+
`~/.local/bin/lightpanda`, the places
|
|
74
|
+
[agent-browser](https://agent-browser.dev/engines/lightpanda) looks too, or
|
|
75
|
+
the binary `LIGHTPANDA_PATH` names in the environment `e2e run` starts in,
|
|
76
|
+
or `binary` in the options. None found fails the run before any test,
|
|
77
|
+
naming the install page.
|
|
78
|
+
Each worker gets its own server, so `workers` sets how many run at once,
|
|
79
|
+
and the run's log names each one:
|
|
80
|
+
|
|
81
|
+
```text
|
|
82
|
+
ℹ lightpanda (1 of 2): lightpanda 1.1.0-nightly at ws://127.0.0.1:51165/
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Attach to a running server
|
|
86
|
+
|
|
87
|
+
A Lightpanda already running, such as the Docker image in CI, is a
|
|
88
|
+
[`connect`](/reference/web#options) target rather than a provider: the
|
|
89
|
+
engine attaches to its CDP endpoint and leaves it running.
|
|
90
|
+
|
|
91
|
+
```ts title="e2e.config.ts"
|
|
92
|
+
web({ connect: { cdpEndpoint: () => process.env.LIGHTPANDA_URL ?? 'ws://127.0.0.1:9222/' } })
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
docker run -d --name lightpanda -p 127.0.0.1:9222:9222 lightpanda/browser:nightly
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Every worker then shares that server. One `lightpanda serve` takes 16
|
|
100
|
+
connections by default; raise its `--cdp-max-connections` above `workers`
|
|
101
|
+
when you run more.
|
|
102
|
+
|
|
103
|
+
## Options
|
|
104
|
+
|
|
105
|
+
| Option | Default | Meaning |
|
|
106
|
+
| --------------- | ------------------------------- | ------- |
|
|
107
|
+
| `binary` | `LIGHTPANDA_PATH`, else found | The binary to start: a path, or a name on `PATH`. |
|
|
108
|
+
| `loadResources` | `[]` | Sub-resources the server fetches: `iframe`, `image`, `stylesheet`. See [Fetch more of the page](#fetch-more-of-the-page). |
|
|
109
|
+
| `args` | `[]` | Further `lightpanda serve` flags, after the ones the provider sets. |
|
|
110
|
+
|
|
111
|
+
The provider leases one server per worker slot for the run; it has no
|
|
112
|
+
`scope: 'attempt'`, since Lightpanda does not expose the default context
|
|
113
|
+
identity a per-attempt lease reattaches by. A server that does not answer
|
|
114
|
+
on its port within ten seconds fails the lease, and so does a binary that
|
|
115
|
+
exits first, with its stderr in the message.
|
|
116
|
+
|
|
117
|
+
## Fetch more of the page
|
|
118
|
+
|
|
119
|
+
`lightpanda serve` requests no sub-resources by default: no images, no
|
|
120
|
+
external stylesheets, no iframes. `loadResources` opts into each, as
|
|
121
|
+
`--load-resources` on the server:
|
|
122
|
+
|
|
123
|
+
```ts
|
|
124
|
+
web({ browser: lightpanda({ loadResources: ['stylesheet', 'iframe'] }) })
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
Inline styles, `<style>` blocks, and the `hidden` attribute already decide
|
|
128
|
+
visibility; `stylesheet` makes an external file's rules count too, so a
|
|
129
|
+
node a linked stylesheet hides is hidden to a query. `image` fetches image
|
|
130
|
+
headers so `load` and `error` reflect the real status.
|
|
131
|
+
|
|
132
|
+
## What works
|
|
133
|
+
|
|
134
|
+
Against the e2e testbed's local suite on a Lightpanda nightly from mid-October
|
|
135
|
+
2026, these pass unchanged:
|
|
136
|
+
|
|
137
|
+
- `app.open`, `app.back`, and `expect(browser).toHaveURL` and `toHaveTitle`.
|
|
138
|
+
- Every `screen` query (`getByRole`, `getByLabel`, `getByText`,
|
|
139
|
+
`getByTestId`, filters, `nth`) and the actions on them: `tap`, `fill`,
|
|
140
|
+
`check`, `select`, `press`, and keyboard input.
|
|
141
|
+
- Polling assertions on text, value, count, and state.
|
|
142
|
+
- `browser.route` mocks and request waits.
|
|
143
|
+
- `browser.setCookies` from a test, after a page is open.
|
|
144
|
+
- `app.screenshot` and the screenshot of a failure. Lightpanda paints a plain
|
|
145
|
+
render of the DOM, not the page as Chrome would show it.
|
|
146
|
+
- Agent steps: `agent.act`, `agent.assert`, `agent.extract`, and
|
|
147
|
+
`agent.waitFor` read the screen tree, which Lightpanda provides in full.
|
|
148
|
+
The testbed's agent suite passes its tree-driven flows (todos, wizard,
|
|
149
|
+
sign-in, hover, context menu, long press, upload); what fails is drawn on a
|
|
150
|
+
canvas, dragged, or below the fold. A judgment on pixels sees the plain
|
|
151
|
+
render, so keep the agent on the tree.
|
|
152
|
+
|
|
153
|
+
A test that asserts on what the user sees, not where it is drawn, moves
|
|
154
|
+
over as is.
|
|
155
|
+
|
|
156
|
+
## What does not work
|
|
157
|
+
|
|
158
|
+
Each of these fails the test with the engine's usual error, not a hang. The
|
|
159
|
+
reasons are Lightpanda's, and its
|
|
160
|
+
[Playwright compatibility issue](https://github.com/lightpanda-io/browser/issues/3076)
|
|
161
|
+
tracks most of them.
|
|
162
|
+
|
|
163
|
+
| Feature | What happens | Why |
|
|
164
|
+
| ------- | ------------ | --- |
|
|
165
|
+
| [Sessions](/authentication) (`test.setup` with `session.save`, `session:` on a test) | The setup passes and the session is saved, but a test that restores it opens signed out. | Cookies set on a context before its first page exists are dropped. Set them from the test with `browser.setCookies` after `app.open` instead. |
|
|
166
|
+
| [Dialogs](/browser#handle-a-dialog) | `browser.onDialog` runs, then fails the test with `ENGINE_FAILURE`: `Page.handleJavaScriptDialog` reports no dialog. | Lightpanda answers `alert`, `confirm` (`false`), and `prompt` (`null`) itself before a client can. |
|
|
167
|
+
| [Downloads](/browser#download-a-file) | `browser.waitForDownload` times out. | No download events over CDP. |
|
|
168
|
+
| [Iframes](/browser#reach-into-an-iframe) | `frameLocator` finds nothing. | Iframes are not loaded by default. `loadResources: ['iframe']` loads them; queries and fills then work, but a listener in the host page on an element the frame created does not fire. |
|
|
169
|
+
| Drags: `screen.swipe`, `locator.swipeTo`, and `browser.mouse` moves with a button down | `ENGINE_FAILURE` naming `Input.setInterceptDrags`. | Playwright needs that command for a drag and Lightpanda has not implemented it. |
|
|
170
|
+
| `browser.setViewport` | The page measures the new size, but a `resize` listener never runs. | No `resize` event is dispatched. |
|
|
171
|
+
| [Media emulation](/reference/web#media-emulation): `colorScheme`, `reducedMotion`, `browser.emulateMedia` | The page keeps reporting `light` and full motion. | `Emulation.setEmulatedMedia` does not change what `matchMedia` answers. |
|
|
172
|
+
| Scrolling: `scrollIntoView`, `scrollUntilVisible`, and an action on a node "outside of the viewport" | The scroll is a no-op, and an actionability wait can time out. | Without layout there is no scroll position to move. The plain render stacks every block in one column, so more of a page sits below the fold than in Chrome, and nothing below it can be reached. |
|
|
173
|
+
| `<canvas>` content: a pin, keypad, or chart drawn with the 2D context | A screenshot shows no drawing, and the canvas takes no space. The agent's pixel tier (`tap_at`, `type_at`, a vision judgment) has nothing to see. | `getContext('2d')` returns a context that records nothing the painter draws. |
|
|
174
|
+
|
|
175
|
+
A Lightpanda release that closes one of these makes the row above go away
|
|
176
|
+
with no change on the e2e side; the engine sends the same CDP commands it
|
|
177
|
+
sends Chrome.
|
|
178
|
+
|
|
179
|
+
## Where it fits
|
|
180
|
+
|
|
181
|
+
Lightpanda suits a suite that is heavy on the agent's reading of the screen
|
|
182
|
+
and light on geometry: smoke flows, forms, navigation, API-backed pages, and
|
|
183
|
+
`agent.act` steps whose checks are text and state. It runs that suite in a
|
|
184
|
+
fraction of Chrome's time and memory, on a box with no Chrome installed.
|
|
185
|
+
|
|
186
|
+
Keep a Chromium target for the rest, and select by target when both are in
|
|
187
|
+
one config:
|
|
188
|
+
|
|
189
|
+
```bash
|
|
190
|
+
e2e run --target lightpanda
|
|
191
|
+
e2e run --target chromium
|
|
192
|
+
```
|