agent-device 0.10.1 → 0.10.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -1
- package/dist/src/331.js +3 -3
- package/dist/src/425.js +1 -0
- package/dist/src/bin.js +28 -28
- package/dist/src/core/dispatch.d.ts +2 -0
- package/dist/src/core/session-surface.d.ts +3 -0
- package/dist/src/core/settings-contract.d.ts +2 -1
- package/dist/src/daemon/app-log-ios.d.ts +2 -1
- package/dist/src/daemon/app-log-process.d.ts +1 -1
- package/dist/src/daemon/app-log.d.ts +1 -1
- package/dist/src/daemon/context.d.ts +2 -0
- package/dist/src/daemon/handlers/interaction-common.d.ts +30 -1
- package/dist/src/daemon/handlers/interaction-read.d.ts +14 -0
- package/dist/src/daemon/handlers/interaction-touch.d.ts +2 -3
- package/dist/src/daemon/handlers/interaction.d.ts +5 -12
- package/dist/src/daemon/handlers/snapshot-capture.d.ts +11 -4
- package/dist/src/daemon/snapshot-processing.d.ts +1 -0
- package/dist/src/daemon/types.d.ts +3 -1
- package/dist/src/daemon.js +39 -39
- package/dist/src/platforms/android/index.d.ts +1 -1
- package/dist/src/platforms/android/input-actions.d.ts +1 -0
- package/dist/src/platforms/android/settings.d.ts +1 -1
- package/dist/src/platforms/ios/apps.d.ts +1 -1
- package/dist/src/platforms/ios/macos-helper.d.ts +69 -0
- package/dist/src/platforms/ios/runner-client.d.ts +1 -1
- package/dist/src/utils/command-schema.d.ts +1 -0
- package/dist/src/utils/interactors.d.ts +1 -1
- package/dist/src/utils/snapshot-lines.d.ts +5 -2
- package/dist/src/utils/snapshot.d.ts +8 -1
- package/dist/src/utils/text-surface.d.ts +19 -0
- package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift +8 -0
- package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift +60 -0
- package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Lifecycle.swift +1 -1
- package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift +4 -0
- package/macos-helper/Package.swift +18 -0
- package/macos-helper/Sources/AgentDeviceMacOSHelper/SnapshotTraversal.swift +543 -0
- package/macos-helper/Sources/AgentDeviceMacOSHelper/main.swift +545 -0
- package/package.json +4 -1
- package/skills/agent-device/SKILL.md +25 -334
- package/skills/agent-device/references/bootstrap-install.md +167 -0
- package/skills/agent-device/references/coordinate-system.md +24 -4
- package/skills/agent-device/references/debugging.md +115 -0
- package/skills/agent-device/references/exploration.md +193 -0
- package/skills/agent-device/references/macos-desktop.md +55 -57
- package/skills/agent-device/references/remote-tenancy.md +56 -47
- package/skills/agent-device/references/verification.md +103 -0
- package/dist/src/274.js +0 -1
- package/dist/src/daemon/handlers/interaction-fill.d.ts +0 -3
- package/dist/src/daemon/handlers/interaction-press.d.ts +0 -3
- package/skills/agent-device/references/batching.md +0 -79
- package/skills/agent-device/references/logs-and-debug.md +0 -113
- package/skills/agent-device/references/perf-metrics.md +0 -53
- package/skills/agent-device/references/permissions.md +0 -70
- package/skills/agent-device/references/session-management.md +0 -101
- package/skills/agent-device/references/snapshot-refs.md +0 -102
- package/skills/agent-device/references/video-recording.md +0 -49
|
@@ -3,346 +3,37 @@ name: agent-device
|
|
|
3
3
|
description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
#
|
|
6
|
+
# agent-device
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
For structured exploratory QA bug hunts and reporting, use [../dogfood/SKILL.md](../dogfood/SKILL.md).
|
|
8
|
+
Use this skill as a router.
|
|
10
9
|
|
|
11
|
-
##
|
|
10
|
+
## QA modes
|
|
12
11
|
|
|
13
|
-
|
|
12
|
+
- Open-ended bug hunt with reporting: use [../dogfood/SKILL.md](../dogfood/SKILL.md).
|
|
13
|
+
- Pass/fail QA from acceptance criteria: stay in this skill, start with [references/bootstrap-install.md](references/bootstrap-install.md), then use the QA loop in [references/exploration.md](references/exploration.md).
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
- Normal interaction flow
|
|
17
|
-
- Debug/crash flow
|
|
18
|
-
- Replay maintenance flow
|
|
19
|
-
2. Run one canonical flow below.
|
|
20
|
-
3. Open references only if blocked.
|
|
15
|
+
## Mental model
|
|
21
16
|
|
|
22
|
-
|
|
17
|
+
- First choose the correct target and open the app or session you want to work on.
|
|
18
|
+
- Then inspect the current UI with `snapshot -i` and pick targets from the actual UI state.
|
|
19
|
+
- Act with `press`, `fill`, `get`, `is`, `wait`, or `find`.
|
|
20
|
+
- Re-snapshot after meaningful UI changes instead of reusing stale refs.
|
|
21
|
+
- End by capturing proof if needed, then `close`.
|
|
23
22
|
|
|
24
|
-
|
|
25
|
-
- Normal UI task: `open` -> `snapshot -i` -> `press/click/fill` -> `diff snapshot -i` -> `close`
|
|
26
|
-
- Debug/crash (iOS/Android): `open <app>` -> `logs clear --restart` -> reproduce -> `network dump` -> `logs path` -> targeted `grep`
|
|
27
|
-
- Replay drift: `replay -u <path>` -> verify updated selectors
|
|
28
|
-
- Remote multi-tenant run: allocate lease -> point client at remote daemon base URL -> run commands with tenant isolation flags -> heartbeat/release lease
|
|
29
|
-
- Device-scope isolation run: set iOS simulator set / Android allowlist -> run selectors within scope only
|
|
30
|
-
- macOS desktop task: run the macOS desktop flow, then open [references/macos-desktop.md](references/macos-desktop.md) if context menus, Finder rows, or desktop-specific snapshot behavior matters
|
|
23
|
+
## Decision rules
|
|
31
24
|
|
|
32
|
-
|
|
25
|
+
- Use plain `snapshot` when you need to verify whether text is visible.
|
|
26
|
+
- Use `snapshot -i` mainly for interactive exploration and choosing refs.
|
|
27
|
+
- Use `fill` to replace text.
|
|
28
|
+
- Use `type` to append text.
|
|
29
|
+
- Prefer `@ref` or selector targeting over raw coordinates.
|
|
30
|
+
- Keep the default loop short: `open` -> explore/act -> optional debug or verify -> `close`.
|
|
33
31
|
|
|
34
|
-
|
|
35
|
-
- iOS local QA in mixed simulator/device environments: run `ensure-simulator` first and pass `--device`, `--udid`, or `--ios-simulator-device-set` on later commands.
|
|
36
|
-
- macOS desktop app automation: use `--platform macos`, or `--platform apple --target desktop` when the caller wants one Apple-family selector path.
|
|
37
|
-
- Android local QA: use `install` or `reinstall` for `.apk`/`.aab` files, then relaunch by installed package name.
|
|
38
|
-
- Android React Native + Metro flows: prefer `open <package> --remote-config <path> --relaunch`.
|
|
39
|
-
- In mixed-device environments, always pin the exact target with `--serial`, `--device`, `--udid`, or an isolation scope.
|
|
40
|
-
- For session-bound automation runs, prefer a pre-bound session/platform instead of repeating selectors on every command: set `AGENT_DEVICE_SESSION`, set `AGENT_DEVICE_PLATFORM`, and the daemon will enforce the shared lock policy across CLI, typed client, and RPC entry points.
|
|
41
|
-
- Use `--session-lock reject|strip` (or `AGENT_DEVICE_SESSION_LOCK`) only when you need to override the default reject behavior. Lock mode applies to nested `batch` steps too.
|
|
32
|
+
## Choose a reference
|
|
42
33
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
agent-device
|
|
49
|
-
agent-device snapshot -i
|
|
50
|
-
agent-device press @e3
|
|
51
|
-
agent-device diff snapshot -i
|
|
52
|
-
agent-device fill @e5 "test"
|
|
53
|
-
agent-device close
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
### 1a) Local iOS Simulator QA Flow
|
|
57
|
-
|
|
58
|
-
```bash
|
|
59
|
-
agent-device ensure-simulator --platform ios --device "iPhone 16" --boot
|
|
60
|
-
agent-device open MyApp --platform ios --device "iPhone 16" --session qa-ios --relaunch
|
|
61
|
-
agent-device snapshot -i
|
|
62
|
-
agent-device press @e3
|
|
63
|
-
agent-device close
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
Use this when a physical iPhone is also connected and you want deterministic simulator-only automation.
|
|
67
|
-
|
|
68
|
-
### 1b) Android React Native + Metro QA Flow
|
|
69
|
-
|
|
70
|
-
```bash
|
|
71
|
-
agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554
|
|
72
|
-
agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch
|
|
73
|
-
agent-device snapshot -i
|
|
74
|
-
agent-device close
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
Do not use `open <apk|aab> --relaunch` on Android. Install/reinstall binaries first, then relaunch by package.
|
|
78
|
-
|
|
79
|
-
### 1c) Session-Bound Automation Flow
|
|
80
|
-
|
|
81
|
-
```bash
|
|
82
|
-
export AGENT_DEVICE_SESSION=qa-ios
|
|
83
|
-
export AGENT_DEVICE_PLATFORM=ios
|
|
84
|
-
export AGENT_DEVICE_SESSION_LOCK=strip
|
|
85
|
-
|
|
86
|
-
agent-device open MyApp --relaunch
|
|
87
|
-
agent-device snapshot -i
|
|
88
|
-
agent-device batch --steps-file /tmp/qa-steps.json --json
|
|
89
|
-
agent-device close
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
Use this for orchestrators that must preserve one bound session/device across many plain CLI calls without a wrapper script. In `strip` mode, conflicting selectors such as `--target`, `--device`, `--udid`, `--serial`, and isolation-scope overrides are ignored instead of retargeting the run.
|
|
93
|
-
|
|
94
|
-
### 1d) Android Emulator Session-Bound Flow
|
|
95
|
-
|
|
96
|
-
```bash
|
|
97
|
-
export AGENT_DEVICE_SESSION=qa-android
|
|
98
|
-
export AGENT_DEVICE_PLATFORM=android
|
|
99
|
-
|
|
100
|
-
agent-device reinstall MyApp /path/to/app-debug.apk --serial emulator-5554
|
|
101
|
-
agent-device --session-lock reject open com.example.myapp --relaunch
|
|
102
|
-
agent-device snapshot -i
|
|
103
|
-
agent-device close --shutdown
|
|
104
|
-
```
|
|
105
|
-
|
|
106
|
-
Use this when an Android emulator session must stay pinned while an agent or test runner issues plain CLI commands over time.
|
|
107
|
-
|
|
108
|
-
### 1e) macOS Desktop Flow
|
|
109
|
-
|
|
110
|
-
```bash
|
|
111
|
-
agent-device open TextEdit --platform macos
|
|
112
|
-
agent-device snapshot -i
|
|
113
|
-
agent-device fill @e3 "desktop smoke test"
|
|
114
|
-
agent-device screenshot /tmp/macos-textedit.png
|
|
115
|
-
agent-device close
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
Use this for host Mac desktop apps. Prefer the Apple runner interaction flow (`open`, `snapshot`, `press`, `click`, `fill`, `scroll`, `back`, `record`, `screenshot`). macOS also supports `clipboard read|write`, `trigger-app-event` when a desktop deep-link template is configured, and only `settings appearance light|dark|toggle` under the `settings` command. Do not rely on mobile-only helpers like `install`, `push`, `logs`, or `network` on macOS.
|
|
119
|
-
Prefer selectors or snapshot refs (`@e...`) over raw x/y commands on macOS because the window origin can move between runs.
|
|
120
|
-
Open [references/macos-desktop.md](references/macos-desktop.md) when you need Finder-style list traversal, context-menu flows, or macOS-specific snapshot expectations.
|
|
121
|
-
|
|
122
|
-
### 2) Debug/Crash Flow
|
|
123
|
-
|
|
124
|
-
```bash
|
|
125
|
-
agent-device open MyApp --platform ios
|
|
126
|
-
agent-device logs clear --restart
|
|
127
|
-
agent-device network dump 25
|
|
128
|
-
agent-device logs path
|
|
129
|
-
```
|
|
130
|
-
|
|
131
|
-
Logging is off by default. Enable only for debugging windows.
|
|
132
|
-
`logs clear --restart` requires an active app session (`open <app>` first).
|
|
133
|
-
|
|
134
|
-
### 3) Replay Maintenance Flow
|
|
135
|
-
|
|
136
|
-
```bash
|
|
137
|
-
agent-device replay -u ./session.ad
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
### 4) Remote Tenant Lease Flow (HTTP JSON-RPC)
|
|
141
|
-
|
|
142
|
-
```bash
|
|
143
|
-
# Client points directly at the remote daemon HTTP base URL.
|
|
144
|
-
export AGENT_DEVICE_DAEMON_BASE_URL=http://mac-host.example:4310
|
|
145
|
-
export AGENT_DEVICE_DAEMON_AUTH_TOKEN=<token>
|
|
146
|
-
|
|
147
|
-
# Allocate lease
|
|
148
|
-
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
|
149
|
-
-H "content-type: application/json" \
|
|
150
|
-
-H "Authorization: Bearer <token>" \
|
|
151
|
-
-d '{"jsonrpc":"2.0","id":"alloc-1","method":"agent_device.lease.allocate","params":{"runId":"run-123","tenantId":"acme","ttlMs":60000}}'
|
|
152
|
-
|
|
153
|
-
# Use lease in tenant-isolated command execution
|
|
154
|
-
agent-device \
|
|
155
|
-
--tenant acme \
|
|
156
|
-
--session-isolation tenant \
|
|
157
|
-
--run-id run-123 \
|
|
158
|
-
--lease-id <lease-id> \
|
|
159
|
-
session list --json
|
|
160
|
-
|
|
161
|
-
# Heartbeat and release
|
|
162
|
-
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
|
163
|
-
-H "content-type: application/json" \
|
|
164
|
-
-H "Authorization: Bearer <token>" \
|
|
165
|
-
-d '{"jsonrpc":"2.0","id":"hb-1","method":"agent_device.lease.heartbeat","params":{"leaseId":"<lease-id>","ttlMs":60000}}'
|
|
166
|
-
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
|
167
|
-
-H "content-type: application/json" \
|
|
168
|
-
-H "Authorization: Bearer <token>" \
|
|
169
|
-
-d '{"jsonrpc":"2.0","id":"rel-1","method":"agent_device.lease.release","params":{"leaseId":"<lease-id>"}}'
|
|
170
|
-
```
|
|
171
|
-
|
|
172
|
-
Notes:
|
|
173
|
-
|
|
174
|
-
- `AGENT_DEVICE_DAEMON_BASE_URL` makes the CLI skip local daemon discovery/startup and call the remote HTTP daemon directly.
|
|
175
|
-
- `AGENT_DEVICE_DAEMON_AUTH_TOKEN` is sent in both the JSON-RPC request token and HTTP auth headers.
|
|
176
|
-
- In remote daemon mode, `--debug` does not tail a local `daemon.log`; inspect logs on the remote host instead.
|
|
177
|
-
|
|
178
|
-
## Command Skeleton (Minimal)
|
|
179
|
-
|
|
180
|
-
### Session and navigation
|
|
181
|
-
|
|
182
|
-
```bash
|
|
183
|
-
agent-device devices
|
|
184
|
-
agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators
|
|
185
|
-
agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234
|
|
186
|
-
agent-device ensure-simulator --device "iPhone 16" --ios-simulator-device-set /tmp/tenant-a/simulators
|
|
187
|
-
agent-device ensure-simulator --device "iPhone 16" --runtime com.apple.CoreSimulator.SimRuntime.iOS-18-4 --ios-simulator-device-set /tmp/tenant-a/simulators --boot
|
|
188
|
-
agent-device open [app|url] [url]
|
|
189
|
-
agent-device open [app] --relaunch
|
|
190
|
-
agent-device close [app]
|
|
191
|
-
agent-device install <app> <path-to-binary>
|
|
192
|
-
agent-device install-from-source <url> [--header "name:value"]
|
|
193
|
-
agent-device reinstall <app> <path-to-binary>
|
|
194
|
-
agent-device session list
|
|
195
|
-
```
|
|
196
|
-
|
|
197
|
-
Use `boot` only as fallback when `open` cannot find/connect to a ready target.
|
|
198
|
-
If the workspace repeats the same selectors or device/session flags, prefer a checked-in `agent-device.json` or `--config <path>` over repeating them inline.
|
|
199
|
-
Environment-level defaults follow the same fields via `AGENT_DEVICE_*` names, so persistent host-specific values belong there rather than in committed project config.
|
|
200
|
-
That includes bound-session defaults such as `sessionLock` / `AGENT_DEVICE_SESSION_LOCK` when automation should consistently reject or strip conflicting device routing flags.
|
|
201
|
-
For Android emulators by AVD name, use `boot --platform android --device <avd-name>`.
|
|
202
|
-
For Android emulators without GUI, add `--headless`.
|
|
203
|
-
Use `--target mobile|tv` with `--platform` (required) to pick phone/tablet vs TV targets (AndroidTV/tvOS).
|
|
204
|
-
For Android React Native + Metro flows, install or reinstall the APK first, then use `open <package> --remote-config <path> --relaunch`; do not use `open <apk|aab> --relaunch`.
|
|
205
|
-
For local iOS QA in mixed simulator/device environments, use `ensure-simulator` and pass `--device` or `--udid` so automation does not attach to a physical device by accident.
|
|
206
|
-
For session-bound automation, prefer `AGENT_DEVICE_SESSION` + `AGENT_DEVICE_PLATFORM`; that bound-session default now enables lock mode automatically.
|
|
207
|
-
|
|
208
|
-
Isolation scoping quick reference:
|
|
209
|
-
|
|
210
|
-
- `--ios-simulator-device-set <path>` scopes iOS simulator discovery + command execution to one simulator set.
|
|
211
|
-
- `--android-device-allowlist <serials>` scopes Android discovery/selection to comma/space separated serials.
|
|
212
|
-
- Scope is applied before selectors (`--device`, `--udid`, `--serial`); out-of-scope selectors fail with `DEVICE_NOT_FOUND`.
|
|
213
|
-
- With iOS simulator-set scope enabled, iOS physical devices are not enumerated.
|
|
214
|
-
- In bound-session `strip` mode, conflicting per-call scope/selectors are ignored and the configured binding is restored for the request. Batch steps still inherit the parent `--platform` when they do not set their own.
|
|
215
|
-
|
|
216
|
-
Simulator provisioning quick reference:
|
|
217
|
-
|
|
218
|
-
- Use `ensure-simulator` to create or reuse a named iOS simulator inside a device set before starting a session.
|
|
219
|
-
- `--device <name>` is required (e.g. `"iPhone 16 Pro"`). `--runtime <id>` pins the runtime; omit to use the newest compatible one.
|
|
220
|
-
- `--boot` boots it immediately. Returns `udid`, `device`, `runtime`, `ios_simulator_device_set`, `created`, `booted`.
|
|
221
|
-
- Idempotent: safe to call repeatedly; reuses an existing matching simulator by default.
|
|
222
|
-
|
|
223
|
-
TV quick reference:
|
|
224
|
-
|
|
225
|
-
- AndroidTV: `open`/`apps` use TV launcher discovery automatically.
|
|
226
|
-
- TV target selection works on emulators/simulators and connected physical devices (AndroidTV + AppleTV).
|
|
227
|
-
- tvOS: runner-driven interactions and snapshots are supported (`snapshot`, `wait`, `press`, `fill`, `get`, `scroll`, `back`, `home`, `app-switcher`, `record` and related selector flows).
|
|
228
|
-
- tvOS `back`/`home`/`app-switcher` map to Siri Remote actions (`menu`, `home`, double-home) in the runner.
|
|
229
|
-
- tvOS follows iOS simulator-only command semantics for helpers like `pinch`, `settings`, and `push`.
|
|
230
|
-
|
|
231
|
-
### Snapshot and targeting
|
|
232
|
-
|
|
233
|
-
```bash
|
|
234
|
-
agent-device snapshot -i
|
|
235
|
-
agent-device diff snapshot -i
|
|
236
|
-
agent-device find "Sign In" click
|
|
237
|
-
agent-device press @e1
|
|
238
|
-
agent-device fill @e2 "text"
|
|
239
|
-
agent-device is visible 'id="anchor"'
|
|
240
|
-
```
|
|
241
|
-
|
|
242
|
-
`press` is canonical tap command; `click` is an alias.
|
|
243
|
-
On macOS, use `click --button secondary <@ref|selector>` to open a context menu before the next `snapshot -i`.
|
|
244
|
-
For desktop-specific heuristics and Finder guidance, see [references/macos-desktop.md](references/macos-desktop.md).
|
|
245
|
-
|
|
246
|
-
### Utilities
|
|
247
|
-
|
|
248
|
-
```bash
|
|
249
|
-
agent-device appstate
|
|
250
|
-
agent-device clipboard read
|
|
251
|
-
agent-device clipboard write "token"
|
|
252
|
-
agent-device keyboard status
|
|
253
|
-
agent-device keyboard dismiss
|
|
254
|
-
agent-device perf --json
|
|
255
|
-
agent-device network dump [limit] [summary|headers|body|all]
|
|
256
|
-
agent-device push <bundle|package> <payload.json|inline-json>
|
|
257
|
-
agent-device trigger-app-event screenshot_taken '{"source":"qa"}'
|
|
258
|
-
agent-device get text @e1
|
|
259
|
-
agent-device screenshot out.png
|
|
260
|
-
agent-device settings permission grant notifications
|
|
261
|
-
agent-device settings permission reset camera
|
|
262
|
-
agent-device trace start
|
|
263
|
-
agent-device trace stop ./trace.log
|
|
264
|
-
```
|
|
265
|
-
|
|
266
|
-
### Batch (when sequence is already known)
|
|
267
|
-
|
|
268
|
-
```bash
|
|
269
|
-
agent-device batch --steps-file /tmp/batch-steps.json --json
|
|
270
|
-
```
|
|
271
|
-
|
|
272
|
-
### Performance Check
|
|
273
|
-
|
|
274
|
-
- Use `agent-device perf --json` (or `metrics --json`) after `open`.
|
|
275
|
-
- For detailed metric semantics, caveats, and interpretation guidance, see [references/perf-metrics.md](references/perf-metrics.md).
|
|
276
|
-
|
|
277
|
-
## Guardrails (High Value Only)
|
|
278
|
-
|
|
279
|
-
- Re-snapshot after UI mutations (navigation/modal/list changes).
|
|
280
|
-
- Prefer `snapshot -i`; scope/depth only when needed.
|
|
281
|
-
- Use refs for discovery, selectors for replay/assertions.
|
|
282
|
-
- `find "<query>" click --json` returns `{ ref, locator, query, x, y }` — all derived from the matched snapshot node. Do not rely on these fields from raw `press`/`click` responses for observability; use `find` instead.
|
|
283
|
-
- Use `fill` for clear-then-type semantics; use `type` for focused append typing.
|
|
284
|
-
- Use `install` for in-place app upgrades (keep app data when platform permits), and `reinstall` for deterministic fresh-state runs.
|
|
285
|
-
- App binary format support for `install`/`reinstall`: Android `.apk`/`.aab`, iOS `.app`/`.ipa`.
|
|
286
|
-
- Android `.aab` requires `bundletool` in `PATH`, or `AGENT_DEVICE_BUNDLETOOL_JAR=<path-to-bundletool-all.jar>` with `java` in `PATH`.
|
|
287
|
-
- Android `.aab` optional: set `AGENT_DEVICE_ANDROID_BUNDLETOOL_MODE=<mode>` to control bundletool `build-apks --mode` (default: `universal`).
|
|
288
|
-
- iOS `.ipa`: extract/install from `Payload/*.app`; when multiple app bundles are present, `<app>` is used as a bundle id/name hint.
|
|
289
|
-
- iOS `appstate` is session-scoped; Android `appstate` is live foreground state. iOS responses include `device_udid` and `ios_simulator_device_set` for isolation verification.
|
|
290
|
-
- iOS `open` responses include `device_udid` and `ios_simulator_device_set` to confirm which simulator handled the session.
|
|
291
|
-
- Clipboard helpers: `clipboard read` / `clipboard write <text>` are supported on macOS, Android, and iOS simulators; iOS physical devices are not supported yet.
|
|
292
|
-
- Android keyboard helpers: `keyboard status|get|dismiss` report keyboard visibility/type and dismiss via keyevent when visible.
|
|
293
|
-
- `network dump` is best-effort and parses HTTP(s) entries from the session app log file.
|
|
294
|
-
- Biometric settings: iOS simulator supports `settings faceid|touchid <match|nonmatch|enroll|unenroll>`; Android supports `settings fingerprint <match|nonmatch>` where runtime tooling is available.
|
|
295
|
-
- For AndroidTV/tvOS selection, always pair `--target` with `--platform` (`ios`, `android`, or `apple` alias); target-only selection is invalid.
|
|
296
|
-
- `push` simulates notification delivery:
|
|
297
|
-
- iOS simulator uses APNs-style payload JSON.
|
|
298
|
-
- Android uses broadcast action + typed extras (string/boolean/number).
|
|
299
|
-
- `trigger-app-event` requires app-defined deep-link hooks and URL template configuration (`AGENT_DEVICE_APP_EVENT_URL_TEMPLATE` or platform-specific variants).
|
|
300
|
-
- On macOS, set `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE` when the desktop app uses a different deep-link template than iOS/Android.
|
|
301
|
-
- `trigger-app-event` requires an active session or explicit selectors (`--platform`, `--device`, `--udid`, `--serial`); on iOS physical devices, custom-scheme triggers require active app context.
|
|
302
|
-
- Canonical trigger behavior and caveats are documented in [`website/docs/docs/commands.md`](../../website/docs/docs/commands.md) under **App event triggers**.
|
|
303
|
-
- Permission settings are app-scoped and require an active session app:
|
|
304
|
-
`settings permission <grant|deny|reset> <camera|microphone|photos|contacts|notifications> [full|limited]`
|
|
305
|
-
- iOS simulator permission alerts: use `alert wait` then `alert accept/dismiss` — `accept`/`dismiss` retry internally for up to 2 s so you do not need manual sleeps. See [references/permissions.md](references/permissions.md).
|
|
306
|
-
- `full|limited` mode applies only to iOS `photos`; other targets reject mode.
|
|
307
|
-
- On Android, non-ASCII `fill/type` may require an ADB keyboard IME on some system images; only install IME APKs from trusted sources and verify checksum/signature.
|
|
308
|
-
- If using `--save-script`, prefer explicit path syntax (`--save-script=flow.ad` or `./flow.ad`).
|
|
309
|
-
- For tenant-isolated remote runs, always pass `--tenant`, `--session-isolation tenant`, `--run-id`, and `--lease-id` together.
|
|
310
|
-
- Use short lease TTLs and heartbeat only while work is active; release leases immediately after run completion/failure.
|
|
311
|
-
- Env equivalents for scoped runs: `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` (compat `IOS_SIMULATOR_DEVICE_SET`) and
|
|
312
|
-
`AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST` (compat `ANDROID_DEVICE_ALLOWLIST`).
|
|
313
|
-
- For explicit remote client mode, prefer `AGENT_DEVICE_DAEMON_BASE_URL` / `--daemon-base-url` instead of relying on local daemon metadata or loopback-only ports.
|
|
314
|
-
|
|
315
|
-
## Common Failure Patterns
|
|
316
|
-
|
|
317
|
-
- `Failed to access Android app sandbox for /path/app-debug.apk`: Android relaunch/runtime-hint flow received an APK path instead of an installed package name. Use `reinstall` first, then `open <package> --relaunch`.
|
|
318
|
-
- `mkdir: Needs 1 argument` while writing `ReactNativeDevPrefs.xml`: likely an older `agent-device` build or stale global install is still using the shell-based Android runtime-hint writer. Verify the exact binary being invoked.
|
|
319
|
-
- `Failed to terminate iOS app`: the flow may have selected a physical iPhone or an unavailable iOS target. Re-run with `ensure-simulator`, then pin the simulator with `--device` or `--udid`.
|
|
320
|
-
|
|
321
|
-
## Security and Trust Notes
|
|
322
|
-
|
|
323
|
-
- Prefer a preinstalled `agent-device` binary over on-demand package execution.
|
|
324
|
-
- If install is required, pin an exact version (for example: `npx --yes agent-device@<exact-version> --help`).
|
|
325
|
-
- Signing/provisioning environment variables are optional, sensitive, and only for iOS physical-device setup.
|
|
326
|
-
- Logs/artifacts are written under `~/.agent-device`; replay scripts write to explicit paths you provide.
|
|
327
|
-
- For remote daemon mode, prefer `AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual` on the host plus client-side `AGENT_DEVICE_DAEMON_BASE_URL`, with `AGENT_DEVICE_HTTP_AUTH_HOOK` and tenant-scoped lease admission where needed.
|
|
328
|
-
- Keep logging off unless debugging and use least-privilege/isolated environments for autonomous runs.
|
|
329
|
-
|
|
330
|
-
## Common Mistakes
|
|
331
|
-
|
|
332
|
-
- Mixing debug flow into normal runs (keep logs off unless debugging).
|
|
333
|
-
- Continuing to use stale refs after screen transitions.
|
|
334
|
-
- Using URL opens with Android `--activity` (unsupported combination).
|
|
335
|
-
- Treating `boot` as default first step instead of fallback.
|
|
336
|
-
|
|
337
|
-
## References
|
|
338
|
-
|
|
339
|
-
- [references/snapshot-refs.md](references/snapshot-refs.md)
|
|
340
|
-
- [references/macos-desktop.md](references/macos-desktop.md)
|
|
341
|
-
- [references/logs-and-debug.md](references/logs-and-debug.md)
|
|
342
|
-
- [references/session-management.md](references/session-management.md)
|
|
343
|
-
- [references/permissions.md](references/permissions.md)
|
|
344
|
-
- [references/video-recording.md](references/video-recording.md)
|
|
345
|
-
- [references/coordinate-system.md](references/coordinate-system.md)
|
|
346
|
-
- [references/batching.md](references/batching.md)
|
|
347
|
-
- [references/perf-metrics.md](references/perf-metrics.md)
|
|
348
|
-
- [references/remote-tenancy.md](references/remote-tenancy.md)
|
|
34
|
+
- Pick target device, install, open, or manage sessions: [references/bootstrap-install.md](references/bootstrap-install.md)
|
|
35
|
+
- Need to discover UI, pick refs, wait, query, or interact: [references/exploration.md](references/exploration.md)
|
|
36
|
+
- Need logs, network, alerts, permissions, or failure triage: [references/debugging.md](references/debugging.md)
|
|
37
|
+
- Need screenshots, diff, recording, replay maintenance, or perf data: [references/verification.md](references/verification.md)
|
|
38
|
+
- Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: [references/macos-desktop.md](references/macos-desktop.md)
|
|
39
|
+
- Need to connect to a remote `agent-device` daemon over HTTP or use tenant leases: [references/remote-tenancy.md](references/remote-tenancy.md)
|
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
# Bootstrap and Install
|
|
2
|
+
|
|
3
|
+
## When to open this file
|
|
4
|
+
|
|
5
|
+
Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting.
|
|
6
|
+
|
|
7
|
+
## Main commands to reach for first
|
|
8
|
+
|
|
9
|
+
- `devices`
|
|
10
|
+
- `ensure-simulator`
|
|
11
|
+
- `open`
|
|
12
|
+
- `install` or `reinstall`
|
|
13
|
+
- `close`
|
|
14
|
+
- `session list`
|
|
15
|
+
|
|
16
|
+
## Most common mistake to avoid
|
|
17
|
+
|
|
18
|
+
Do not start acting before you have pinned the correct target and opened an `app` session. In mixed-device environments, always pass `--device`, `--udid`, or `--serial`.
|
|
19
|
+
|
|
20
|
+
## Canonical loop
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
agent-device ensure-simulator --platform ios --device "iPhone 17 Pro" --boot
|
|
24
|
+
agent-device open MyApp --platform ios --device "iPhone 17 Pro" --relaunch
|
|
25
|
+
agent-device snapshot -i
|
|
26
|
+
agent-device close
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
## Choose the right starting point
|
|
30
|
+
|
|
31
|
+
- iOS local QA: prefer simulators unless the task explicitly requires physical hardware.
|
|
32
|
+
- iOS in mixed simulator and device environments: run `ensure-simulator` first, then keep using `--device` or `--udid`.
|
|
33
|
+
- TV targets: use `--target tv` together with `--platform` when the task is for tvOS or Android TV rather than phone or tablet surfaces.
|
|
34
|
+
- Android binary flow: use `install` or `reinstall` for `.apk` or `.aab`, then open by installed package name.
|
|
35
|
+
- Android React Native plus Metro flow: `reinstall <app> <apk>` first, then `open <package> --remote-config <path> --relaunch`.
|
|
36
|
+
- macOS desktop app flow: use `open <app> --platform macos`. Only load [macos-desktop.md](macos-desktop.md) if a desktop surface or macOS-specific behavior matters.
|
|
37
|
+
|
|
38
|
+
TV example:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
agent-device open MyTvApp --platform ios --target tv
|
|
42
|
+
agent-device open com.example.androidtv --platform android --target tv
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Session rules
|
|
46
|
+
|
|
47
|
+
- Use `--session <name>` when you need a named session:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
agent-device --session auth open Settings --platform ios
|
|
51
|
+
agent-device --session auth snapshot -i
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
- Use `open <app>` before interactions.
|
|
55
|
+
- Use `close` when done. Add `--shutdown` when you want simulators or emulators torn down with the session.
|
|
56
|
+
- Use semantic session names when you need multiple concurrent runs.
|
|
57
|
+
- Use `--save-script=<path>` on `close` when you want to keep a replay script.
|
|
58
|
+
- For dev loops where state can linger, prefer `open <app> --relaunch`.
|
|
59
|
+
- In iOS sessions, use `open <app>` for the app itself. Use `open <url>` for deep links, and `open <app> <url>` when you need to launch the app and deep link in one step.
|
|
60
|
+
- On iOS, `appstate` is session-scoped and requires the matching active session on the target device.
|
|
61
|
+
|
|
62
|
+
## After a session is established
|
|
63
|
+
|
|
64
|
+
Once you have opened the correct session on the correct target, default to the conservative rule: keep the session binding on follow-up commands, and stop repeating device-routing flags unless you are intentionally retargeting.
|
|
65
|
+
|
|
66
|
+
- Prefer `--session <name>` on follow-up commands, or use sandboxed `AGENT_DEVICE_SESSION`.
|
|
67
|
+
- Do not keep repeating `--platform`, `--target`, `--device`, `--udid`, `--serial`, or similar target-selection flags on normal follow-up commands.
|
|
68
|
+
- Only omit follow-up session flags when the environment explicitly guarantees isolation.
|
|
69
|
+
|
|
70
|
+
Good shared-host pattern:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro"
|
|
74
|
+
agent-device --session auth snapshot -i
|
|
75
|
+
agent-device --session auth press @e3
|
|
76
|
+
agent-device --session auth close
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Bad shared-host pattern:
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro"
|
|
83
|
+
agent-device --session auth snapshot -i --platform ios --device "iPhone 17 Pro"
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Use target-selection flags again only when you are choosing the target before opening a session, or when you explicitly mean to retarget.
|
|
87
|
+
|
|
88
|
+
## Session-bound automation
|
|
89
|
+
|
|
90
|
+
Use this when an orchestrator must keep plain CLI calls on one session and device.
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
export AGENT_DEVICE_SESSION=qa-ios
|
|
94
|
+
export AGENT_DEVICE_PLATFORM=ios
|
|
95
|
+
export AGENT_DEVICE_SESSION_LOCK=strip
|
|
96
|
+
|
|
97
|
+
agent-device open MyApp --relaunch
|
|
98
|
+
agent-device snapshot -i
|
|
99
|
+
agent-device close
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
- `AGENT_DEVICE_SESSION` plus `AGENT_DEVICE_PLATFORM` provides the default binding.
|
|
103
|
+
- `--session-lock reject|strip` controls whether conflicting per-call routing flags fail or are ignored.
|
|
104
|
+
- Conflicts include explicit retargeting flags such as `--platform`, `--target`, `--device`, `--udid`, `--serial`, `--ios-simulator-device-set`, and `--android-device-allowlist`.
|
|
105
|
+
- Lock policy applies to nested `batch` steps too.
|
|
106
|
+
- Compatibility aliases remain supported: `--session-locked`, `--session-lock-conflicts`, `AGENT_DEVICE_SESSION_LOCKED`, and `AGENT_DEVICE_SESSION_LOCK_CONFLICTS`.
|
|
107
|
+
|
|
108
|
+
Android emulator variant:
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
export AGENT_DEVICE_SESSION=qa-android
|
|
112
|
+
export AGENT_DEVICE_PLATFORM=android
|
|
113
|
+
|
|
114
|
+
agent-device reinstall MyApp /path/to/app-debug.apk --serial emulator-5554
|
|
115
|
+
agent-device --session-lock reject open com.example.myapp --relaunch
|
|
116
|
+
agent-device snapshot -i
|
|
117
|
+
agent-device close --shutdown
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## Scoped discovery
|
|
121
|
+
|
|
122
|
+
Use scoped discovery when one run must not see host-global device lists.
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators
|
|
126
|
+
agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
- Scope is applied before `--device`, `--udid`, and `--serial`.
|
|
130
|
+
- Out-of-scope selectors fail with `DEVICE_NOT_FOUND`.
|
|
131
|
+
- With iOS simulator-set scope enabled, iOS physical devices are not enumerated.
|
|
132
|
+
- If the scoped iOS simulator set is empty, the error should point at the set path and suggest creating a simulator in that set.
|
|
133
|
+
- Environment equivalents:
|
|
134
|
+
- `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET`
|
|
135
|
+
- `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST`
|
|
136
|
+
|
|
137
|
+
## Session inspection and replay
|
|
138
|
+
|
|
139
|
+
```bash
|
|
140
|
+
agent-device session list
|
|
141
|
+
agent-device replay ./session.ad --session auth
|
|
142
|
+
agent-device replay -u ./session.ad --session auth
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
- iOS session entries include `device_udid` and `ios_simulator_device_set`. Use them to confirm routing in concurrent runs.
|
|
146
|
+
- Prefer selector-based actions and assertions in saved replay scripts.
|
|
147
|
+
- Tenant isolation namespaces sessions as `<tenant>:<session>` during tenant-scoped runs.
|
|
148
|
+
|
|
149
|
+
## When to leave this file
|
|
150
|
+
|
|
151
|
+
- Once the correct target and session are pinned, move to [exploration.md](exploration.md).
|
|
152
|
+
- If opening, startup, permissions, or logs become the blocker, switch to [debugging.md](debugging.md).
|
|
153
|
+
|
|
154
|
+
## Install and open examples
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554
|
|
158
|
+
agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
Do not use `open <apk|aab> --relaunch` on Android.
|
|
162
|
+
|
|
163
|
+
## Security and trust notes
|
|
164
|
+
|
|
165
|
+
- Treat signing, provisioning, and daemon auth values as host secrets. Do not paste them into shared logs or commit them to source control.
|
|
166
|
+
- Prefer Xcode Automatic Signing over manual overrides when a physical iOS device is involved.
|
|
167
|
+
- Keep persistent host-specific defaults in environment variables rather than checked-in project config.
|
|
@@ -1,8 +1,28 @@
|
|
|
1
1
|
# Coordinate System
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
## When to open this file
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
- Units: device points for iOS, pixels for Android
|
|
5
|
+
Open this file only when you must use raw coordinates instead of selectors or `@ref` targeting.
|
|
7
6
|
|
|
8
|
-
|
|
7
|
+
## Main commands to reach for first
|
|
8
|
+
|
|
9
|
+
- `screenshot`
|
|
10
|
+
- coordinate-based `click` or `swipe`
|
|
11
|
+
|
|
12
|
+
## Most common mistake to avoid
|
|
13
|
+
|
|
14
|
+
Do not assume coordinates mean the same thing across platforms or runs. Prefer selectors and refs first.
|
|
15
|
+
|
|
16
|
+
## Canonical loop
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
agent-device screenshot /tmp/current-screen.png
|
|
20
|
+
agent-device click 120 240
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## Rules
|
|
24
|
+
|
|
25
|
+
- Origin is the top-left of the device screen.
|
|
26
|
+
- iOS uses device points.
|
|
27
|
+
- Android uses pixels.
|
|
28
|
+
- Use screenshots to reason about coordinates before acting.
|