e2e 0.15.1 → 0.15.2

package/docs/mobile.mdx

@@ -0,0 +1,555 @@
+---
+title: Testing iOS and Android
+description: Set up a simulator or emulator target, write a device test, and know what differs from the web.
+---
+
+`@e2e-dev/mobile` runs tests on iOS simulators and Android emulators. It is
+built on [agent-device](https://github.com/callstack/agent-device).
+Tests use the same `agent`, `screen`, and `expect` APIs as browser tests.
+A test launches the configured app with `app.open()`; between tests the app
+stays where the last one left it.
+
+## Set up
+
+Install Xcode with an iOS simulator runtime, or the Android SDK with an
+emulator. Check your setup with:
+
+<CodeGroup>
+```bash npm
+npx agent-device doctor
+```
+
+```bash pnpm
+pnpm exec agent-device doctor
+```
+
+```bash bun
+bunx agent-device doctor
+```
+</CodeGroup>
+
+Follow the [Quickstart](/quickstart) and choose **Mobile (iOS/Android)**.
+The wizard selects iOS on macOS and Android elsewhere. It starts with the
+Settings app and one worker:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { mobile } from '@e2e-dev/mobile';
+
+export default {
+  // Replace Settings with your app's bundle id.
+  targets: [{ name: 'ios', engine: mobile({ platform: 'ios' }), app: { bundleId: 'Settings' } }],
+  workers: 1,
+} satisfies E2EConfig;
+```
+
+The runner boots the device before test timing starts. Boot time does not
+count toward `launchTimeout`.
+
+## Point at your app
+
+The app under test is the target's `app`; `mobile()` only drives it.
+Replace `Settings` with your app's bundle id or package name in
+`app.bundleId`: it is the app `app.open()`, `app.restart()`, and
+`app.clearState()` launch. Add `appPath` to install a build first:
+
+```ts
+{
+  name: 'ios',
+  engine: mobile({ platform: 'ios' }),
+  app: {
+    bundleId: 'com.example.app',
+    appPath: './build/MyApp.app', // the iOS .app bundle
+  },
+}
+```
+
+The engine installs nothing on its own: `app.appPath` names the build, and
+the suite installs it where it says so, once per device, with
+`device.installApp()` (no path installs `app.appPath`), for example in a
+fixture every test takes. Use an `.app` bundle for iOS or an `.apk` for
+Android. Without `bundleId`, the app the build installs is the one
+`app.open()` launches. A device target needs one of the two.
+
+To run the suite in CI on GitHub Actions, EAS Workflows, Bitrise, or
+Codemagic, see [Mobile CI](/mobile-ci).
+
+### Launch arguments and permissions
+
+`app.launchArguments` reach the app on every fresh launch the engine performs
+(`app.open()`, `app.restart()`, `app.clearState()`): the process arguments on
+iOS, `am start` arguments on Android. `app.permissions` are set before each of
+those launches, so the app starts holding them and no system prompt appears;
+`app.clearState()` puts them back after it reset them with the data. `deny`
+differs by platform: iOS records the refusal and the app is told no without
+a prompt, Android revokes the permission and the OS asks again when the app
+requests it. The warm-up in `prepare` starts the iOS runner with
+agent-device's `prepare ios-runner`, whose startup budget covers a cold
+simulator, then opens the app once with neither, to put the worker's session
+on the app; the process a test sees comes from `app.open()`. A build the
+suite installs itself gets the runner start only, so the first test's
+`app.open()` does not pay for it. Arguments that select a build mode change
+what the app is: give each mode its own `app.identity` so their recordings
+stay apart.
+
+```ts
+app: {
+  bundleId: 'com.example.app',
+  launchArguments: ['-e2e', 'YES'],
+  permissions: { camera: 'grant', notifications: 'deny', location: 'reset' },
+},
+```
+
+One test launches with its own through `device.openApp`:
+
+```ts
+await device.openApp('com.example.app', { relaunch: true, permissions: { camera: 'deny' } });
+```
+
+## Several devices
+
+Add a target for each platform:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { mobile } from '@e2e-dev/mobile';
+
+const app = { bundleId: 'com.example.app' };
+
+export default {
+  targets: [
+    { name: 'iphone', engine: mobile({ platform: 'ios' }), app },
+    { name: 'pixel', engine: mobile({ platform: 'android' }), app },
+  ],
+  workers: 1,
+} satisfies E2EConfig;
+```
+
+Tests run on both targets. Use `platforms: ['ios']` or
+`platforms: ['android']` for tests with platform-specific labels or behavior.
+
+To run files in parallel, provide a device pool and set `workers` to at
+least the pool's size. Each device gets one worker:
+
+```ts
+mobile({ platform: 'ios', device: ['iPhone 17', 'iPhone 17 Pro'] })
+```
+
+Without `device`, the engine uses booted devices up to the worker limit.
+If none are booted, it boots one. Two workers never drive the same device.
+
+## Hosted devices
+
+A `device` can also be a `DeviceProvider`: an object that leases one hosted
+device per worker slot when the run starts and releases it when the run
+ends. The engine drives each lease through the agent-device daemon the lease
+names instead of the local one, so any service that runs an agent-device
+daemon next to a simulator, an emulator, or a physical device works. Nothing
+in the framework knows a vendor: for [EAS Simulators](/integrations/eas),
+`easSimulators()` from `@e2e-dev/eas` is the provider, and for
+any other service a provider is a few dozen lines in your project.
+
+```ts
+export interface DeviceProvider {
+  /** Label in progress lines and error messages. */
+  readonly name: string;
+  /** Called once per worker slot, for every slot at once, in the runner process. */
+  acquire(request: DeviceRequest): Promise<DeviceLease>;
+  /** Called once per lease when the run ends, on every exit path. */
+  release(lease: DeviceLease, context: DeviceReleaseContext): Promise<void>;
+  /** Optional: the service's own recording of the device, in the worker, for an attempt that records video. */
+  record?(lease: DeviceLease, context: ProviderRecordContext): Promise<ProviderRecording>;
+}
+
+export interface DeviceLease {
+  /** The provider's handle (a session id), handed back to `release`. */
+  readonly id: string;
+  /** The agent-device daemon the worker connects to; a lease names this, `client`, or both. */
+  readonly daemon?: { readonly baseUrl: string; readonly authToken?: string };
+  /** agent-device client configuration the worker's client is created with: a lease scope, a `stateDir`. */
+  readonly client?: DeviceClientConfig;
+  /** Device to select inside that daemon, when it hosts more than one. */
+  readonly device?: string;
+  /** Device to select inside that daemon by the id agent-device lists it under: a simulator UDID or an Android serial. */
+  readonly deviceId?: string;
+  /** Bundle id or package the provider installed itself; skips the target's `app.appPath`. */
+  readonly installedApp?: string;
+}
+```
+
+`acquire` receives the platform, run id, target, slot, the target's
+`app.bundleId` as `app` and its resolved `app.appPath` as `appPath`, the
+agent-device version the engine's client speaks (`agentDeviceVersion`, for a
+service that starts a daemon of a chosen version), the directory the config's
+relative paths resolve against (`projectRoot`), the run's environment, an
+interrupt signal, and a `log` that reaches the reporter. Read tokens from
+`request.env`, not `process.env`, and project files under
+`request.projectRoot`, not `process.cwd()`. A target with nothing to run leases nothing. Every lease
+granted is held by the engine, one it then refuses included: a slot that
+fails to lease ends the run before any test, and the leases that succeeded
+are still released. Release happens after the last worker exits, on every
+exit path including failure, interrupt, and a `prepare` that threw, within
+`cleanupTimeout`. `release` receives the object `acquire` returned, so a
+provider can keep its own fields on it; only `id`, `daemon`, `client`,
+`device`, `deviceId`, and `installedApp` travel to the workers. Report `installedApp` only for a
+request that carried `appPath`.
+Leased devices are warmed like local ones: booted, the iOS runner started,
+and opened once before the run's clock starts.
+
+`client` is agent-device client configuration, JSON data the worker's client
+is created with on top of the session name and the `daemon` address (the
+engine owns `session`, `daemonBaseUrl`, `daemonAuthToken`, and
+`daemonTransport`, and refuses a lease that sets them). It is how a device reached through agent-device's own
+device-cloud runtimes is told apart: the daemon resolves a hosted device from
+the lease scope on each command (`tenant`, `runId`, `leaseId`, `leaseBackend`,
+`leaseProvider`, as `leases.allocate` returned it), and a command without one
+falls back to the local inventory. A `stateDir` in it points the worker at a
+daemon the provider started, so such a provider needs no `daemon` address at
+all.
+
+A service that records its devices can record the attempt's video too.
+Implement `record` and an attempt that records video calls it from the
+worker, with the lease that worker drives, instead of starting agent-device's
+screen recording. It returns a `ProviderRecording` (import the type from
+`e2e/engine`), stopped when the attempt's video ends: `stop` either writes a
+file into the directory it is given and returns `{ file }`, kept as
+`video/<file>` beside the attempt's other artifacts, or returns
+`{ url, mediaType }` for a recording the service keeps, which the report
+links to. The lease `record` gets is the lease as it traveled to the worker,
+so a field only `release` needs is not on it. A `record` that fails fails
+the attempt's launch, named after the provider and the lease; a `stop` that
+fails is tried once more when the attempt ends. A provider that cannot
+record leaves `record` out, and the engine records with agent-device.
+
+A provider against a typical session API has three requests: create a
+session, poll it until the daemon is reachable, stop it.
+
+```ts title="device-provider.ts"
+import type { DeviceProvider } from '@e2e-dev/mobile';
+
+/**
+ * Leases devices from a service that starts an agent-device daemon per
+ * session. Replace the three requests with your service's API.
+ */
+export function hostedDevices(options: { serviceUrl: string; image: string }): DeviceProvider {
+  return {
+    name: 'hosted-devices',
+    async acquire(request) {
+      const token = request.env['DEVICE_SERVICE_TOKEN'];
+      if (token === undefined || token === '') throw new Error('DEVICE_SERVICE_TOKEN is not set');
+      const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' };
+      // 1. Ask for a device; name the session after the run so it can be found later.
+      const created = await fetch(`${options.serviceUrl}/sessions`, {
+        method: 'POST',
+        headers,
+        body: JSON.stringify({
+          platform: request.platform,
+          image: options.image,
+          name: `e2e ${request.runId} ${request.targetName} ${request.slot + 1}/${request.slots}`,
+        }),
+        signal: request.signal,
+      });
+      if (!created.ok) throw new Error(`device service: HTTP ${created.status}`);
+      const { id } = (await created.json()) as { id: string };
+      // 2. Wait until the session exposes its agent-device daemon.
+      for (;;) {
+        const response = await fetch(`${options.serviceUrl}/sessions/${id}`, { headers, signal: request.signal });
+        const session = (await response.json()) as { status: string; daemonUrl?: string; daemonToken?: string; viewerUrl?: string };
+        if (session.status === 'ready' && session.daemonUrl !== undefined) {
+          if (session.viewerUrl !== undefined) request.log(`watch at ${session.viewerUrl}`);
+          return { id, daemon: { baseUrl: session.daemonUrl, authToken: session.daemonToken } };
+        }
+        if (session.status !== 'starting') throw new Error(`device service: session ${id} is ${session.status}`);
+        await new Promise((resolve) => setTimeout(resolve, 5_000));
+      }
+    },
+    async release(lease, context) {
+      // 3. Stop the session; the run waits for this within cleanupTimeout.
+      const token = context.env['DEVICE_SERVICE_TOKEN'] ?? '';
+      await fetch(`${options.serviceUrl}/sessions/${lease.id}/stop`, {
+        method: 'POST',
+        headers: { Authorization: `Bearer ${token}` },
+        signal: context.signal,
+      });
+    },
+  };
+}
+```
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { mobile } from '@e2e-dev/mobile';
+import { hostedDevices } from './device-provider.ts';
+
+export default {
+  targets: [
+    {
+      name: 'ios',
+      engine: mobile({
+        platform: 'ios',
+        device: hostedDevices({ serviceUrl: 'https://devices.example.com', image: 'ios-18-iphone-17' }),
+      }),
+      app: { bundleId: 'com.example.app' },
+    },
+  ],
+  workers: 2,
+} satisfies E2EConfig;
+```
+
+Two workers lease two sessions in parallel. Hosted sessions are billed while
+they run, so keep `workers` at what the suite can fill. The replay cache keys
+on `app.bundleId`, so traces recorded locally replay on hosted devices.
+
+A device cloud agent-device speaks to itself needs no session API of your
+own: the provider allocates a lease through a daemon it starts for the run and
+hands the worker the lease scope as `client`.
+
+```ts title="device-cloud-provider.ts"
+import { createAgentDeviceClient } from 'agent-device';
+import type { DeviceProvider } from '@e2e-dev/mobile';
+
+/**
+ * Leases devices from a device cloud agent-device itself speaks to, through
+ * a daemon started for the run. The daemon's runtime for the cloud creates
+ * the instance when the lease is allocated and deletes it when the lease is
+ * released; the worker's client carries the lease, so every command lands on
+ * that instance. `provider` is agent-device's name for the cloud; its runtime
+ * reads the credentials from the environment the daemon starts with.
+ */
+export function cloudDevices(options: { provider: string; stateDir: string }): DeviceProvider {
+  const scopes = new Map<string, { tenant: string; runId: string; leaseId: string; leaseBackend: 'ios-instance' | 'android-instance'; leaseProvider: string }>();
+  return {
+    name: options.provider,
+    async acquire(request) {
+      // One daemon per run, under a directory of its own.
+      const stateDir = `${options.stateDir}/${request.runId}`;
+      const client = createAgentDeviceClient({ stateDir, session: `lease-${request.slot}` });
+      const leaseBackend = request.platform === 'ios' ? ('ios-instance' as const) : ('android-instance' as const);
+      const lease = await client.leases.allocate({
+        tenant: options.provider,
+        runId: request.runId,
+        leaseBackend,
+        leaseProvider: options.provider,
+        platform: request.platform,
+        target: 'mobile',
+      });
+      request.log(`leased ${lease.leaseId}`);
+      const scope = { tenant: lease.tenantId, runId: lease.runId, leaseId: lease.leaseId, leaseBackend, leaseProvider: options.provider };
+      scopes.set(lease.leaseId, scope);
+      // The worker's client is created with these fields; the daemon resolves the device from them.
+      return { id: lease.leaseId, client: { stateDir, ...scope } };
+    },
+    async release(lease) {
+      const scope = scopes.get(lease.id);
+      if (scope === undefined) return;
+      const client = createAgentDeviceClient({ stateDir: `${options.stateDir}/${scope.runId}`, session: 'release' });
+      await client.leases.release(scope);
+    },
+  };
+}
+```
+
+## Writing a device test
+
+`screen` maps native controls to common roles such as `button`, `textbox`,
+`listitem`, and `switch`. Use `getByTestId` for native element identifiers.
+
+```ts
+import { test } from '@e2e-dev/mobile';
+import { expect } from 'e2e';
+
+test('shows the version offline in dark mode', async ({ agent, app, device, screen }) => {
+  await device.setAppearance('dark');
+  await device.setNetwork('offline');
+  await app.open();
+  await agent.act('go to General, then About');
+  await expect(screen.getByRole('button', /^iOS Version/)).toBeVisible();
+  await expect(device.locator('role=NavigationBar id=About')).toBeVisible();
+});
+```
+
+`selectOption`, `setInputFiles`, `scrollIntoView`, `secondaryTap`, and
+`modifiers` on `tap` or `doubleTap` are unsupported on devices. Use taps and swipes to interact with native controls.
+
+### The `device` fixture
+
+Use `device` to manage network state, permissions, appearance, app installs,
+and other device settings without a model call. Each call appears as a
+`device.<method>` step in the report.
+
+`device.locator('...')` accepts an agent-device selector for controls that
+`screen` queries cannot identify:
+
+```ts
+await expect(device.locator('role=NavigationBar id=About')).toBeVisible();
+await device.locator('id=SW_VERSION_SPECIFIER role=Button').tap();
+```
+
+Use `device.installApp` within a test to check upgrades or fresh installs; with no path it installs the target's `app.appPath`:
+
+```ts
+const { app } = await device.installApp('./build/MyApp.app', { reinstall: true });
+await device.openApp(app);
+```
+
+`device.openApp` takes a bundle id, package, or display name, never a link.
+`device.openLink` opens a deep link or a web link into the pinned app, or
+into the `app` named in its options, for magic-link sign-in and deep-link
+routes:
+
+```ts
+await device.openLink('myapp://orders/42');
+await expect(screen.getByText('Order #42')).toBeVisible();
+```
+
+`device.clearKeychain()` resets the simulator's keychain, which
+`app.clearState()` leaves alone, so a stored session token is gone on the
+next launch. iOS only; Android has no keychain to reset.
+
+`device.fold(pose)` puts a foldable iOS simulator, the iPhone Duo, in a hinge
+pose: `closed` lights the outer display, `half-open` (a 130° book) and `open`
+the inner one. It returns once the simulator reports the hinge at that angle,
+typically 10 to 16 seconds, so keep `actionTimeout` at 20 seconds or more, and
+the next observation reads the app on the other display. The engine never
+resets the pose: a retry and the next test start in whatever pose a test left,
+so fold back in `afterEach`, which runs when the test fails too:
+
+```ts
+import { afterEach, test } from '@e2e-dev/mobile';
+
+afterEach(async ({ device }) => {
+  await device.fold('closed');
+});
+
+test('the list and the detail split across the fold', async ({ agent, app, device }) => {
+  await app.open();
+  await device.fold('half-open');
+  await agent.assert('the list and the detail sit on either side of the fold', { vision: true });
+});
+```
+
+Every method is in the [mobile engine reference](/reference/mobile#device).
+
+### Agent tools
+
+The agent already has basic actions such as `tap`, `type`, and `scroll`.
+Add `mobileTools` for app switching, device gestures, and system alerts:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { mobile } from '@e2e-dev/mobile';
+import { mobileTools } from '@e2e-dev/mobile/tools';
+import { gateway } from 'ai';
+
+const iphone = mobile({ platform: 'ios' });
+const pixel = mobile({ platform: 'android' });
+const app = { bundleId: 'com.example.app' };
+
+export default {
+  targets: [
+    { name: 'iphone', engine: iphone, app },
+    { name: 'pixel', engine: pixel, app },
+  ],
+  workers: 1,
+  agents: {
+    default: {
+      model: gateway('openai/gpt-6-luna-fast'),
+      tools: mobileTools(iphone, pixel),
+    },
+  },
+} satisfies E2EConfig;
+```
+
+Pass each configured device engine to `mobileTools`. Calls go to the
+engine running the current attempt. `open_app`, `swipe`, and `alert` all
+change device state, so the agent takes over for them during cache replay.
+The [mobile engine reference](/reference/mobile#agent-tools) describes each
+tool.
+
+## What differs from the web
+
+| On the web | On a device |
+| --- | --- |
+| `app.open('/path')` and the agent's `navigate` verb | No app URL. `app.open()` launches the target's `app.bundleId` fresh, `device.openLink(url)` opens a deep link or web link into it, and there is no `navigate` |
+| `test.setup` saves a session, `session:` restores it | Not available. A simulator has no portable session snapshot; sign in per test or seed the app |
+| Secret input | `type_secret` and `screen.getByLabel(...).fill(secret)` fill declared secrets into secure fields |
+| Playwright trace per attempt | No Playwright trace; screenshots mask secure fields |
+| `--video` writes a WebM screencast | `--video` writes an MP4 of the device screen with taps shown (`videoTouches: false` hides them), or keeps a device provider's own recording |
+| The cache anchors on the page path | The cache anchors on the location `<bundle id> / <screen title>`, the title read off the navigation bar on iOS and the toolbar on Android |
+
+### Known device limitations
+
+- `placeholder` is the only attribute a device node has.
+- A React Native tab (`accessibilityRole="tab"`) on iOS has role `other`.
+  Query it by test id or label and assert `toBeSelected`. Checkboxes,
+  radios, and `toBeChecked` work on both platforms.
+- On iOS a plain React Native `View` has no children in the tree, so
+  `filter({ has })` on it and queries scoped to it match nothing. Scope to the screen's `ScrollView`
+  (`device.locator('role=ScrollView')`), or give the children test ids.
+- An element hidden with `display: none` is absent from the tree, so
+  `toBeAttached` fails. Assert
+  `toHaveCount(0)` or `toBeHidden`.
+- `doubleTap()` on iOS is two taps about 285 ms apart. Give a double-tap
+  detector a window of 350 ms or more.
+- On iOS, `device.foregroundApp()` still names the session's app after
+  `device.home()`. Check the screen instead.
+- On an iPhone Duo, `--video` records the display that was lit when the
+  recording started. After `device.fold` moves the app to the other display,
+  the rest of the video is black.
+
+## Troubleshooting
+
+A device failure shows agent-device's own message. Under the agent, an
+engine fault with no more specific code reports as `APP_UNREACHABLE`, so
+read the message before the code: the app is often not the part that
+failed.
+
+**The iOS automation runner is busy, wedged, or could not present a
+snapshot.** A heavy or animating screen can overrun the runner, and a run
+stopped with Ctrl-C can leave it busy for the next run. The message reads:
+
+```text
+APP_UNREACHABLE: agent.act failed: snapshot failed: the iOS automation runner is still
+finishing a command that overran its watchdog (session e2e-ios-0 on iPhone 17 Pro):
+The iOS runner is still finishing a previous command that exceeded its execution
+watchdog (usually an accessibility capture on a heavy or animating screen). The app
+is fine. Wait a few seconds and rerun. If the next run meets it again, stop the
+daemon (`npx agent-device daemon stop`) and reboot the simulator
+(`xcrun simctl shutdown <udid>`, then `boot`).
+```
+
+Do what it says.
+
+**A timeout on the device.** A simulator boot, runner connection, or app
+launch that times out on the device is `ENGINE_FAILURE` with agent-device's
+message and hint, never one of e2e's own timeout codes (`LAUNCH_TIMEOUT`,
+`ACTION_FAILED`, `STEP_TIMEOUT`).
+
+<CodeGroup>
+```bash npm
+npx agent-device doctor          # the toolchain, the daemon, the booted devices
+npx agent-device daemon stop     # after a wedge: drops every session, the next run starts the daemon
+xcrun simctl shutdown <udid> && xcrun simctl boot <udid>
+```
+
+```bash pnpm
+pnpm exec agent-device doctor          # the toolchain, the daemon, the booted devices
+pnpm exec agent-device daemon stop     # after a wedge: drops every session, the next run starts the daemon
+xcrun simctl shutdown <udid> && xcrun simctl boot <udid>
+```
+
+```bash bun
+bunx agent-device doctor          # the toolchain, the daemon, the booted devices
+bunx agent-device daemon stop     # after a wedge: drops every session, the next run starts the daemon
+xcrun simctl shutdown <udid> && xcrun simctl boot <udid>
+```
+</CodeGroup>
+
+<CardGroup cols={2}>
+  <Card title="Mobile engine reference" icon="mobile-screen" href="/reference/mobile">
+    Every option, the `device` fixture, and the tool pack.
+  </Card>
+  <Card title="Writing tests" icon="pen-to-square" href="/writing-tests">
+    Goals, checks, and the deterministic APIs, on any platform.
+  </Card>
+</CardGroup>