e2e 0.15.1 → 0.16.0

package/docs/mobile-ci.mdx

@@ -0,0 +1,567 @@
+---
+title: Mobile CI
+description: Run iOS and Android tests on GitHub Actions, EAS Workflows, Bitrise, and Codemagic.
+---
+
+A mobile CI job does the same four things on every service: build the app
+for a simulator or emulator, boot a device, install the build, and run
+`npx e2e run`. Only the syntax changes. Read
+[Continuous integration](/ci) first for what CI mode changes, model
+credentials, sharding, and exit codes; everything there applies here.
+
+## What the job needs
+
+| | iOS | Android |
+| --- | --- | --- |
+| Machine | macOS with Xcode and a simulator runtime | Linux with KVM (nested virtualization) |
+| Build | Release simulator `.app`, no signing | Release `.apk` |
+| Device | A booted simulator | A booted emulator |
+| Node | 22.12 or later | 22.12 or later |
+
+Build Release on both platforms: a React Native or Expo debug build loads
+its JavaScript from Metro, which no CI job here starts, so the app opens on
+an error screen. The examples assume a React Native project with `ios/` and
+`android/` committed; an Expo project without them runs `npx expo prebuild`
+first.
+
+The config pins the app by its bundle id or package name, and the job
+installs the build before the run:
+
+```ts title="e2e.config.ts"
+import type { E2EConfig } from 'e2e';
+import { mobile } from '@e2e-dev/mobile';
+
+export default {
+  targets: [
+    { name: 'ios', engine: mobile({ platform: 'ios' }), app: { bundleId: 'com.example.app' } },
+    { name: 'android', engine: mobile({ platform: 'android' }), app: { bundleId: 'com.example.app' } },
+  ],
+  workers: 1,
+} satisfies E2EConfig;
+```
+
+<CodeGroup>
+```bash npm
+# iOS
+xcrun simctl install booted path/to/MyApp.app
+npx e2e run --target ios --reporter list,junit
+
+# Android
+adb install -r path/to/app.apk
+npx e2e run --target android --reporter list,junit
+```
+
+```bash pnpm
+# iOS
+xcrun simctl install booted path/to/MyApp.app
+pnpm exec e2e run --target ios --reporter list,junit
+
+# Android
+adb install -r path/to/app.apk
+pnpm exec e2e run --target android --reporter list,junit
+```
+
+```bash bun
+# iOS
+xcrun simctl install booted path/to/MyApp.app
+bunx e2e run --target ios --reporter list,junit
+
+# Android
+adb install -r path/to/app.apk
+bunx e2e run --target android --reporter list,junit
+```
+</CodeGroup>
+
+The engine uses the booted device and asks agent-device to boot one when
+none is up. Boot it yourself as a step anyway: a fresh Linux runner has no
+emulator image to boot until you install one, and a boot in its own step
+tells a slow or failed boot apart from a failed test. To have the suite
+install the build instead, set `app.appPath` and call `device.installApp()`; see
+[Point at your app](/mobile#point-at-your-app).
+
+A few things hold on every service:
+
+- **One job per platform.** `--target ios` on a Mac, `--target android` on
+  a Linux machine with KVM. Hosted Macs run on Apple Silicon, which cannot
+  nest virtualization, so the Android emulator does not start there.
+- **Pin the iOS runtime.** A simulator picked by name (`iPhone 17`) runs
+  whichever iOS the image defaults to, and the accessibility tree changes
+  between iOS releases: a runner image update can flip a locator assertion
+  with no change in your code. The examples create a simulator on a named
+  runtime and boot it by UDID; replace `iOS-26-5` with a runtime the image
+  ships (`xcrun simctl list runtimes`) and bump it on purpose.
+- **The iOS runner is compiled on first use.** agent-device builds its
+  XCUITest runner into `~/.agent-device` the first time it drives a
+  simulator. Cache that directory where the service lets you.
+- **Upload `.e2e/` whatever the outcome.** `report.json` and `junit.xml` say
+  what failed; `.e2e/artifacts/` holds the screenshots and screen dumps that
+  say why, and the video when you run with `--video`.
+- **Replay what you recorded locally.** [Commit the replay cache](/cache#commit-the-replay-cache)
+  so `agent.act` steps replay without a model call. Agent judgments
+  (`agent.assert`, `agent.waitFor`, `agent.extract`) always call the model.
+  The cache keys on `app`, so a recording made on your simulator replays on
+  the CI one.
+- **Keep the model key from untrusted code.** A job that runs a pull
+  request's code hands the key to that code, its tests and dependencies
+  included. Expose it only to builds of branches you trust; see
+  [Model credentials](/ci#model-credentials) and the
+  [security model](/security).
+
+## GitHub Actions
+
+Two jobs, one per platform. The iOS job builds with `xcodebuild` for the
+simulator; swap in your own build command as long as it writes a simulator
+`.app` without launching it. For Expo, `npx expo run:ios --configuration
+Release --device generic --no-bundler --output build/ios` builds without
+booting a simulator and writes the `.app` under `build/ios`.
+
+```yaml title=".github/workflows/e2e-mobile.yml"
+name: e2e mobile
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  ios:
+    runs-on: macos-26
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 26
+          cache: npm
+      - run: npm ci
+      - run: pod install
+        working-directory: ios
+
+      - name: Build for the simulator
+        run: |
+          xcodebuild -workspace ios/MyApp.xcworkspace -scheme MyApp \
+            -configuration Release -sdk iphonesimulator \
+            -destination 'generic/platform=iOS Simulator' \
+            -derivedDataPath build
+      - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ~/.agent-device
+          key: agent-device-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
+
+      - name: Boot a simulator and install the app
+        run: |
+          UDID=$(xcrun simctl create e2e "iPhone 17" com.apple.CoreSimulator.SimRuntime.iOS-26-5)
+          xcrun simctl bootstatus "$UDID" -b
+          xcrun simctl install "$UDID" build/Build/Products/Release-iphonesimulator/MyApp.app
+
+      - run: npx e2e run --target ios --reporter list,junit
+        env:
+          AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}
+
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: ${{ !cancelled() }}
+        with:
+          name: e2e-ios
+          path: .e2e
+          if-no-files-found: warn
+
+  android:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 26
+          cache: npm
+      - uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1
+        with:
+          distribution: temurin
+          java-version: 17
+      - run: npm ci
+
+      - name: Enable KVM
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
+      # React Native and Expo templates sign release builds with the debug keystore.
+      - name: Build the APK
+        run: cd android && ./gradlew :app:assembleRelease
+
+      - name: Run on an emulator
+        uses: reactivecircus/android-emulator-runner@a421e43855164a8197daf9d8d40fe71c6996bb0d # v2.38.0
+        with:
+          api-level: 36
+          arch: x86_64
+          target: google_apis
+          profile: pixel_7
+          emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
+          disable-animations: true
+          script: |
+            adb install -r android/app/build/outputs/apk/release/app-release.apk
+            npx e2e run --target android --reporter list,junit
+        env:
+          AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}
+
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: ${{ !cancelled() }}
+        with:
+          name: e2e-android
+          path: .e2e
+          if-no-files-found: warn
+```
+
+The emulator runner action boots the emulator, runs each `script` line in
+its own shell, and shuts the emulator down when the step ends, so the install
+and the run belong in the same step. Add the [GitHub reporter](/github) to
+post each job's results on the pull request.
+
+This repository runs its own mobile suites this way; see
+[`mobile.yml`](https://github.com/tester-army/e2e/blob/main/.github/workflows/mobile.yml)
+for a version that caches the Expo build per native fingerprint, boots two
+simulators for two workers, and reruns failures with `--last-failed`.
+
+## EAS Workflows
+
+A `build` job produces the simulator `.app` or the `.apk` on EAS, and a
+custom job downloads it, boots a device, and runs the suite. Add a build
+profile that needs no credentials:
+
+```json title="eas.json"
+{
+  "build": {
+    "e2e": {
+      "withoutCredentials": true,
+      "ios": { "simulator": true },
+      "android": { "buildType": "apk" }
+    }
+  }
+}
+```
+
+The iOS job runs on a macOS runner; the Android one needs a
+`nested-virtualization` Linux runner, the only EAS machines that can run the
+emulator:
+
+```yaml title=".eas/workflows/e2e.yml"
+name: e2e
+
+on:
+  pull_request:
+    branches: ['*']
+  push:
+    branches: [main]
+
+defaults:
+  tools:
+    node: '22'
+
+jobs:
+  build_ios:
+    type: build
+    params:
+      platform: ios
+      profile: e2e
+
+  build_android:
+    type: build
+    params:
+      platform: android
+      profile: e2e
+
+  e2e_ios:
+    needs: [build_ios]
+    runs_on: macos-medium
+    environment: preview
+    env:
+      CI: '1'
+    steps:
+      - uses: eas/checkout
+      - uses: eas/install_node_modules
+      - uses: eas/download_build
+        id: download_build
+        with:
+          build_id: ${{ needs.build_ios.outputs.build_id }}
+          extensions: [app]
+      - name: Boot a simulator and install the app
+        run: |
+          UDID=$(xcrun simctl create e2e "iPhone 17" com.apple.CoreSimulator.SimRuntime.iOS-26-5)
+          xcrun simctl bootstatus "$UDID" -b
+          xcrun simctl install "$UDID" "${{ steps.download_build.outputs.artifact_path }}"
+      - name: Run e2e
+        run: npx e2e run --target ios --reporter list,junit
+      - uses: eas/upload_artifact
+        if: ${{ always() }}
+        with:
+          type: other
+          name: e2e-ios
+          path: .e2e
+
+  e2e_android:
+    needs: [build_android]
+    runs_on: linux-medium-nested-virtualization
+    environment: preview
+    env:
+      CI: '1'
+    steps:
+      - uses: eas/checkout
+      - uses: eas/install_node_modules
+      - uses: eas/download_build
+        id: download_build
+        with:
+          build_id: ${{ needs.build_android.outputs.build_id }}
+          extensions: [apk]
+      - uses: eas/start_android_emulator
+        with:
+          system_image_package: system-images;android-35;google_apis;x86_64
+      - name: Install the app
+        run: adb install -r "${{ steps.download_build.outputs.artifact_path }}"
+      - name: Run e2e
+        run: npx e2e run --target android --reporter list,junit
+      - uses: eas/upload_artifact
+        if: ${{ always() }}
+        with:
+          type: other
+          name: e2e-android
+          path: .e2e
+```
+
+- **Secrets.** Store `AI_GATEWAY_API_KEY` (or your provider's key) as an
+  [EAS environment variable](https://docs.expo.dev/eas/environment-variables/)
+  with secret visibility. `environment:` picks which EAS environment a job
+  reads; set it explicitly, since custom jobs default to `production`.
+- **CI mode.** `CI: '1'` turns on the [CI defaults](/ci#what-ci-changes)
+  whether or not the runner sets it.
+- **Emulator function.** `eas/start_android_emulator` is an EAS build
+  function, documented under
+  [custom builds](https://docs.expo.dev/custom-builds/schema/) rather than
+  on the Workflows syntax page. It downloads the system image you name with
+  `sdkmanager`. The iOS job creates its simulator with `xcrun simctl` to pin
+  the runtime.
+- **Rebuilding less.** A `fingerprint` job with `get-build` and `repack`
+  reuses the last native build and only rebundles the JavaScript. The
+  [agent-device EAS template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml)
+  wires this up for both platforms.
+
+EAS Workflows also has a built-in `maestro` job type that boots the device
+and runs Maestro flows. For e2e, use a custom job as above.
+
+## Bitrise
+
+One workflow per platform, grouped into a pipeline. Bitrise ships steps for
+booting both devices: `xcode-start-simulator` on macOS, and `avd-manager`
+with `wait-for-android-emulator` on Linux. Start the emulator before the
+Gradle build so it boots while the APK compiles.
+
+```yaml title="bitrise.yml"
+format_version: '25'
+default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git
+
+tools:
+  nodejs: 22:installed
+
+pipelines:
+  e2e:
+    triggers:
+      push:
+      - branch: main
+      pull_request:
+      - target_branch: main
+    workflows:
+      e2e-ios: {}
+      e2e-android: {}
+
+workflows:
+  e2e-ios:
+    meta:
+      bitrise.io:
+        stack: osx-xcode-26.6.x
+    steps:
+    - git-clone@8: {}
+    - script@1:
+        title: Install dependencies
+        inputs:
+        - content: |-
+            #!/usr/bin/env bash
+            set -euo pipefail
+            npm ci
+            cd ios && pod install
+    - xcode-build-for-simulator@3:
+        inputs:
+        - project_path: ios/MyApp.xcworkspace
+        - scheme: MyApp
+        - configuration: Release
+    - xcode-start-simulator@0:
+        inputs:
+        - destination: platform=iOS Simulator,name=iPhone 17,OS=26.5
+    - script@1:
+        title: Run e2e
+        inputs:
+        - content: |-
+            #!/usr/bin/env bash
+            set -euo pipefail
+            xcrun simctl install booted "$BITRISE_APP_DIR_PATH"
+            npx e2e run --target ios --reporter list,junit
+    - script@1:
+        title: Package the e2e output
+        is_always_run: true
+        inputs:
+        - content: |-
+            #!/usr/bin/env bash
+            if [ -d .e2e ]; then (cd .e2e && zip -qr "$BITRISE_DEPLOY_DIR/e2e-ios.zip" .); fi
+    - custom-test-results-export@1:
+        inputs:
+        - test_name: e2e-ios
+        - base_path: $BITRISE_SOURCE_DIR/.e2e/junit.xml
+        - search_pattern: '*'
+    - deploy-to-bitrise-io@2: {}
+
+  e2e-android:
+    meta:
+      bitrise.io:
+        stack: ubuntu-noble-24.04-bitrise-2025-android
+    steps:
+    - git-clone@8: {}
+    - script@1:
+        title: Install dependencies
+        inputs:
+        - content: npm ci
+    - avd-manager@2:
+        inputs:
+        - api_level: 35
+        - tag: google_apis
+        - abi: x86_64
+    - android-build@1:
+        inputs:
+        - project_location: $BITRISE_SOURCE_DIR/android
+        - module: app
+        - variant: release
+        - build_type: apk
+    - wait-for-android-emulator@1: {}
+    - script@1:
+        title: Run e2e
+        inputs:
+        - content: |-
+            #!/usr/bin/env bash
+            set -euo pipefail
+            adb install -r "$BITRISE_APK_PATH"
+            npx e2e run --target android --reporter list,junit
+    - script@1:
+        title: Package the e2e output
+        is_always_run: true
+        inputs:
+        - content: |-
+            #!/usr/bin/env bash
+            if [ -d .e2e ]; then (cd .e2e && zip -qr "$BITRISE_DEPLOY_DIR/e2e-android.zip" .); fi
+    - custom-test-results-export@1:
+        inputs:
+        - test_name: e2e-android
+        - base_path: $BITRISE_SOURCE_DIR/.e2e/junit.xml
+        - search_pattern: '*'
+    - deploy-to-bitrise-io@2: {}
+```
+
+- **Artifacts.** `deploy-to-bitrise-io` uploads only the top level of
+  `$BITRISE_DEPLOY_DIR`, so the workflow zips `.e2e/` into it.
+  `custom-test-results-export` hands `junit.xml` to the Test Reports tab.
+  Both, and `deploy-to-bitrise-io`, run after a failed step.
+- **Secrets.** Add the model key under Secrets in the app's workflow editor.
+  Pull request builds get a secret only with **Expose for Pull Requests**
+  turned on, and builds of a public app never get one: the deterministic
+  suite still runs there, but every agent judgment and every `agent.act`
+  with no recording fails.
+- **Release signing.** A release APK needs signing; React Native and Expo
+  templates sign it with the debug keystore.
+
+## Codemagic
+
+Android runs on a Linux instance. The Ubuntu image comes with AVDs named `emulator` (API 34),
+`emulator-35`, and `emulator-36`.
+
+```yaml title="codemagic.yaml"
+workflows:
+  e2e-ios:
+    name: e2e iOS
+    instance_type: mac_mini_m2
+    max_build_duration: 60
+    environment:
+      groups:
+        - e2e
+      node: 22
+      xcode: latest
+    triggering:
+      events:
+        - push
+        - pull_request
+    scripts:
+      - name: Install dependencies
+        script: npm ci
+      - name: Install pods
+        script: cd ios && pod install
+      - name: Build for the simulator
+        script: |
+          xcodebuild -workspace ios/MyApp.xcworkspace -scheme MyApp \
+            -configuration Release -sdk iphonesimulator \
+            -destination 'generic/platform=iOS Simulator' \
+            -derivedDataPath ios/build
+      - name: Boot a simulator and install the app
+        script: |
+          UDID=$(xcrun simctl create e2e "iPhone 17" com.apple.CoreSimulator.SimRuntime.iOS-26-5)
+          xcrun simctl bootstatus "$UDID" -b
+          xcrun simctl install "$UDID" ios/build/Build/Products/Release-iphonesimulator/MyApp.app
+      - name: Run e2e
+        script: npx e2e run --target ios --reporter list,junit
+        test_report: .e2e/junit.xml
+    artifacts:
+      - .e2e/report.json
+      - .e2e/junit.xml
+      - .e2e/artifacts/**
+
+  e2e-android:
+    name: e2e Android
+    instance_type: linux_x2
+    max_build_duration: 60
+    environment:
+      groups:
+        - e2e
+      node: 22
+    triggering:
+      events:
+        - push
+        - pull_request
+    scripts:
+      - name: Install dependencies
+        script: npm ci
+      - name: Build the APK
+        script: cd android && ./gradlew :app:assembleRelease
+      - name: Boot the emulator and install the app
+        script: |
+          emulator -avd emulator-35 -no-window -no-audio -no-boot-anim > /dev/null 2>&1 &
+          adb wait-for-device
+          until [ "$(adb shell getprop sys.boot_completed | tr -d '\r')" = "1" ]; do sleep 2; done
+          adb install -r android/app/build/outputs/apk/release/app-release.apk
+      - name: Run e2e
+        script: npx e2e run --target android --reporter list,junit
+        test_report: .e2e/junit.xml
+    artifacts:
+      - .e2e/report.json
+      - .e2e/junit.xml
+      - .e2e/artifacts/**
+```
+
+Put the model key in the `e2e` environment variable group, marked secret.
+`test_report` feeds `junit.xml` to the build's Tests tab. Linux instances
+need billing enabled.
+
+<CardGroup cols={2}>
+  <Card title="Continuous integration" icon="gears" href="/ci">
+    CI defaults, sharding, reruns, and exit codes.
+  </Card>
+  <Card title="Hosted devices" icon="cloud" href="/mobile#hosted-devices">
+    Lease devices from a service instead of booting them on the runner.
+  </Card>
+</CardGroup>