agent-device 0.8.6 → 0.10.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-device",
-  "version": "0.8.6",
+  "version": "0.10.0",
   "description": "Unified control plane for physical and virtual devices via an agent-driven CLI.",
   "license": "MIT",
   "author": "Callstack",
@@ -23,8 +23,9 @@
     "build": "rslib build",
     "clean:daemon": "rm -f ~/.agent-device/daemon.json && rm -f ~/.agent-device/daemon.lock",
     "build:node": "pnpm build && pnpm clean:daemon",
-    "build:xcuitest": "pnpm build:xcuitest:ios",
+    "build:xcuitest": "pnpm build:xcuitest:ios && pnpm build:xcuitest:macos",
     "build:xcuitest:ios": "rm -rf ~/.agent-device/ios-runner/derived/device && xcodebuild build-for-testing -project ios-runner/AgentDeviceRunner/AgentDeviceRunner.xcodeproj -scheme AgentDeviceRunner -destination \"generic/platform=iOS Simulator\" -derivedDataPath ~/.agent-device/ios-runner/derived",
+    "build:xcuitest:macos": "rm -rf ~/.agent-device/ios-runner/derived/macos && xcodebuild build-for-testing -project ios-runner/AgentDeviceRunner/AgentDeviceRunner.xcodeproj -scheme AgentDeviceRunner -destination \"platform=macOS,arch=$(uname -m)\" -derivedDataPath ~/.agent-device/ios-runner/derived/macos CODE_SIGNING_ALLOWED=NO CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY=\"\" DEVELOPMENT_TEAM=\"\" COMPILER_INDEX_STORE_ENABLE=NO ENABLE_CODE_COVERAGE=NO",
     "build:xcuitest:tvos": "rm -rf ~/.agent-device/ios-runner/derived/tvos && xcodebuild build-for-testing -project ios-runner/AgentDeviceRunner/AgentDeviceRunner.xcodeproj -scheme AgentDeviceRunner -destination \"generic/platform=tvOS Simulator\" -derivedDataPath ~/.agent-device/ios-runner/derived/tvos",
     "build:all": "pnpm build:node && pnpm build:xcuitest",
     "ad": "node bin/agent-device.mjs",

package/skills/agent-device/SKILL.md

@@ -1,9 +1,9 @@
 ---
 name: agent-device
-description: Automates interactions for iOS simulators/devices and Android emulators/devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info on mobile targets.
+description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets.
 ---
 
-# Mobile Automation with agent-device
+# Apple and Android Automation with agent-device
 
 For exploration, use snapshot refs. For deterministic replay, use selectors.
 For structured exploratory QA bug hunts and reporting, use [../dogfood/SKILL.md](../dogfood/SKILL.md).
@@ -22,18 +22,20 @@ Use this skill as a router, not a full manual.
 ## Decision Map
 
 - No target context yet: `devices` -> pick target -> `open`.
-- Normal UI task: `open` -> `snapshot -i` -> `press/fill` -> `diff snapshot -i` -> `close`
-- Debug/crash: `open <app>` -> `logs clear --restart` -> reproduce -> `network dump` -> `logs path` -> targeted `grep`
+- Normal UI task: `open` -> `snapshot -i` -> `press/click/fill` -> `diff snapshot -i` -> `close`
+- Debug/crash (iOS/Android): `open <app>` -> `logs clear --restart` -> reproduce -> `network dump` -> `logs path` -> targeted `grep`
 - Replay drift: `replay -u <path>` -> verify updated selectors
 - Remote multi-tenant run: allocate lease -> point client at remote daemon base URL -> run commands with tenant isolation flags -> heartbeat/release lease
 - Device-scope isolation run: set iOS simulator set / Android allowlist -> run selectors within scope only
+- macOS desktop task: run the macOS desktop flow, then open [references/macos-desktop.md](references/macos-desktop.md) if context menus, Finder rows, or desktop-specific snapshot behavior matters
 
 ## Target Selection Rules
 
 - iOS local QA: use simulators unless the task explicitly requires a physical device.
 - iOS local QA in mixed simulator/device environments: run `ensure-simulator` first and pass `--device`, `--udid`, or `--ios-simulator-device-set` on later commands.
+- macOS desktop app automation: use `--platform macos`, or `--platform apple --target desktop` when the caller wants one Apple-family selector path.
 - Android local QA: use `install` or `reinstall` for `.apk`/`.aab` files, then relaunch by installed package name.
-- Android React Native + Metro flows: set runtime hints with `runtime set` before `open <package> --relaunch`.
+- Android React Native + Metro flows: prefer `open <package> --remote-config <path> --relaunch`.
 - In mixed-device environments, always pin the exact target with `--serial`, `--device`, `--udid`, or an isolation scope.
 - For session-bound automation runs, prefer a pre-bound session/platform instead of repeating selectors on every command: set `AGENT_DEVICE_SESSION`, set `AGENT_DEVICE_PLATFORM`, and the daemon will enforce the shared lock policy across CLI, typed client, and RPC entry points.
 - Use `--session-lock reject|strip` (or `AGENT_DEVICE_SESSION_LOCK`) only when you need to override the default reject behavior. Lock mode applies to nested `batch` steps too.
@@ -67,8 +69,7 @@ Use this when a physical iPhone is also connected and you want deterministic sim
 
 ```bash
 agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554
-agent-device runtime set --session qa-android --platform android --metro-host 10.0.2.2 --metro-port 8081
-agent-device open com.example.myapp --platform android --serial emulator-5554 --session qa-android --relaunch
+agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch
 agent-device snapshot -i
 agent-device close
 ```
@@ -104,6 +105,20 @@ agent-device close --shutdown
 
 Use this when an Android emulator session must stay pinned while an agent or test runner issues plain CLI commands over time.
 
+### 1e) macOS Desktop Flow
+
+```bash
+agent-device open TextEdit --platform macos
+agent-device snapshot -i
+agent-device fill @e3 "desktop smoke test"
+agent-device screenshot /tmp/macos-textedit.png
+agent-device close
+```
+
+Use this for host Mac desktop apps. Prefer the Apple runner interaction flow (`open`, `snapshot`, `press`, `click`, `fill`, `scroll`, `back`, `record`, `screenshot`). macOS also supports `clipboard read|write`, `trigger-app-event` when a desktop deep-link template is configured, and only `settings appearance light|dark|toggle` under the `settings` command. Do not rely on mobile-only helpers like `install`, `push`, `logs`, or `network` on macOS.
+Prefer selectors or snapshot refs (`@e...`) over raw x/y commands on macOS because the window origin can move between runs.
+Open [references/macos-desktop.md](references/macos-desktop.md) when you need Finder-style list traversal, context-menu flows, or macOS-specific snapshot expectations.
+
 ### 2) Debug/Crash Flow
 
 ```bash
@@ -186,7 +201,7 @@ That includes bound-session defaults such as `sessionLock` / `AGENT_DEVICE_SESSI
 For Android emulators by AVD name, use `boot --platform android --device <avd-name>`.
 For Android emulators without GUI, add `--headless`.
 Use `--target mobile|tv` with `--platform` (required) to pick phone/tablet vs TV targets (AndroidTV/tvOS).
-For Android React Native + Metro flows, install or reinstall the APK first, set runtime hints with `runtime set`, then use `open <package> --relaunch`; do not use `open <apk|aab> --relaunch`.
+For Android React Native + Metro flows, install or reinstall the APK first, then use `open <package> --remote-config <path> --relaunch`; do not use `open <apk|aab> --relaunch`.
 For local iOS QA in mixed simulator/device environments, use `ensure-simulator` and pass `--device` or `--udid` so automation does not attach to a physical device by accident.
 For session-bound automation, prefer `AGENT_DEVICE_SESSION` + `AGENT_DEVICE_PLATFORM`; that bound-session default now enables lock mode automatically.
 
@@ -225,6 +240,8 @@ agent-device is visible 'id="anchor"'
 ```
 
 `press` is canonical tap command; `click` is an alias.
+On macOS, use `click --button secondary <@ref|selector>` to open a context menu before the next `snapshot -i`.
+For desktop-specific heuristics and Finder guidance, see [references/macos-desktop.md](references/macos-desktop.md).
 
 ### Utilities
 
@@ -271,7 +288,7 @@ agent-device batch --steps-file /tmp/batch-steps.json --json
 - iOS `.ipa`: extract/install from `Payload/*.app`; when multiple app bundles are present, `<app>` is used as a bundle id/name hint.
 - iOS `appstate` is session-scoped; Android `appstate` is live foreground state. iOS responses include `device_udid` and `ios_simulator_device_set` for isolation verification.
 - iOS `open` responses include `device_udid` and `ios_simulator_device_set` to confirm which simulator handled the session.
-- Clipboard helpers: `clipboard read` / `clipboard write <text>` are supported on Android and iOS simulators; iOS physical devices are not supported yet.
+- Clipboard helpers: `clipboard read` / `clipboard write <text>` are supported on macOS, Android, and iOS simulators; iOS physical devices are not supported yet.
 - Android keyboard helpers: `keyboard status|get|dismiss` report keyboard visibility/type and dismiss via keyevent when visible.
 - `network dump` is best-effort and parses HTTP(s) entries from the session app log file.
 - Biometric settings: iOS simulator supports `settings faceid|touchid <match|nonmatch|enroll|unenroll>`; Android supports `settings fingerprint <match|nonmatch>` where runtime tooling is available.
@@ -280,6 +297,7 @@ agent-device batch --steps-file /tmp/batch-steps.json --json
   - iOS simulator uses APNs-style payload JSON.
   - Android uses broadcast action + typed extras (string/boolean/number).
 - `trigger-app-event` requires app-defined deep-link hooks and URL template configuration (`AGENT_DEVICE_APP_EVENT_URL_TEMPLATE` or platform-specific variants).
+- On macOS, set `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE` when the desktop app uses a different deep-link template than iOS/Android.
 - `trigger-app-event` requires an active session or explicit selectors (`--platform`, `--device`, `--udid`, `--serial`); on iOS physical devices, custom-scheme triggers require active app context.
 - Canonical trigger behavior and caveats are documented in [`website/docs/docs/commands.md`](../../website/docs/docs/commands.md) under **App event triggers**.
 - Permission settings are app-scoped and require an active session app:
@@ -319,6 +337,7 @@ agent-device batch --steps-file /tmp/batch-steps.json --json
 ## References
 
 - [references/snapshot-refs.md](references/snapshot-refs.md)
+- [references/macos-desktop.md](references/macos-desktop.md)
 - [references/logs-and-debug.md](references/logs-and-debug.md)
 - [references/session-management.md](references/session-management.md)
 - [references/permissions.md](references/permissions.md)

package/skills/agent-device/references/macos-desktop.md

@@ -0,0 +1,89 @@
+# macOS Desktop Automation
+
+Use this reference for host Mac apps such as Finder, TextEdit, System Settings, Preview, or browser apps running as normal desktop windows.
+
+## Mental model
+
+- `snapshot -i` should describe UI that is visible to a human in the current front window.
+- Context menus are not ambient UI. Open them explicitly with `click --button secondary`, then re-snapshot.
+- Prefer refs for exploration and selectors for deterministic replay/assertions.
+- Avoid raw `x y` coordinates unless refs/selectors are impossible.
+
+## Canonical flow
+
+```bash
+agent-device open Finder --platform macos
+agent-device snapshot -i
+agent-device click @e66 --button secondary --platform macos
+agent-device snapshot -i
+agent-device close
+```
+
+## What to expect from snapshots
+
+- `snapshot -i` prioritizes visible window content over dormant menu infrastructure.
+- File rows, sidebar items, toolbar controls, search fields, and visible context menus should appear.
+- Finder and other native apps may expose duplicate-looking structures such as row wrapper nodes, `cell` nodes, and child `text` or `text-field` nodes.
+- Treat those as distinct AX nodes unless you have a stronger selector anchor.
+
+## Context menus
+
+Use secondary click when the app exposes actions only through the contextual menu.
+
+```bash
+agent-device click @e66 --button secondary --platform macos
+agent-device snapshot -i
+```
+
+Expected pattern:
+
+1. Snapshot visible content.
+2. Secondary-click the target row/item.
+3. Snapshot again.
+4. Interact with newly visible `menu-item` nodes.
+
+Do not expect context-menu items to appear before the menu is opened.
+
+## Finder-specific guidance
+
+- `snapshot -i` should still expose visible folder rows even when nothing is selected.
+- Unselected folder contents should still be visible in `snapshot -i` through list/table rows.
+- A file row may expose multiple nodes with the same label, including a row container, name cell, and child text/text-field.
+- For opening a context menu, prefer the outer visible row/cell ref over a nested text child if both exist.
+- After secondary click, expect actions such as `Rename`, `Quick Look`, `Copy`, `Compress`, and tag-related items in the next snapshot.
+
+## Raw snapshots
+
+Use `snapshot --raw` only when debugging AX structure or collector issues.
+
+```bash
+agent-device snapshot --raw --platform macos
+```
+
+- Raw output is larger and less token-efficient.
+- It is useful for verifying whether missing UI is absent from the AX tree or only filtered from interactive output.
+- Do not use raw output as the default agent loop when `snapshot -i` already shows the visible window content you need.
+
+## Selector guidance
+
+Good macOS selectors usually anchor on one of:
+
+- `label="Downloads"`
+- `label="failed-step.json"`
+- `role=button label="Search"`
+- `role=menu-item label="Rename"`
+
+Prefer exact labels when the desktop UI is stable. Use `id=...` when the AX identifier is clearly app-owned and not a framework-generated `_NS:*` value.
+
+## Things not to rely on
+
+- Mobile-only helpers like `install`, `reinstall`, `push`, `logs`, `network`, or generic `alert`
+- Long-press as a substitute for right-click
+- Raw coordinate assumptions across runs; macOS windows can move
+- Framework-generated `_NS:*` identifiers as stable selectors
+
+## Troubleshooting
+
+- If visible window content is missing from `snapshot -i`, re-snapshot once after the UI settles.
+- If the wrong menu opened or no menu appeared, retry secondary-clicking the row/cell wrapper instead of the nested text node.
+- If the app has multiple windows, ensure the correct one is frontmost before relying on refs.

package/skills/agent-device/references/snapshot-refs.md

@@ -1,9 +1,10 @@
-# Snapshot Refs and Selectors (Mobile)
+# Snapshot Refs and Selectors
 
 ## Purpose
 
 Refs are useful for discovery/debugging. For deterministic scripts, use selectors.
 For tap interactions, `press` is canonical; `click` is an equivalent alias.
+For host Mac desktop apps, pair this reference with [macos-desktop.md](macos-desktop.md) because context menus and native list/table structures need desktop-specific handling.
 
 ## Snapshot
 
@@ -29,6 +30,13 @@ agent-device press @e2
 agent-device fill @e5 "test"
 ```
 
+On macOS, if actions live in a context menu, use:
+
+```bash
+agent-device click @e5 --button secondary --platform macos
+agent-device snapshot -i
+```
+
 ## Using selectors (deterministic)
 
 ```bash
@@ -71,7 +79,8 @@ Efficient pattern:
 ## Troubleshooting
 
 - Ref not found: re-snapshot.
-- If XCTest returns 0 nodes, foreground app state may have changed. Re-open the app or retry after state is stable.
+- If `snapshot` returns 0 nodes, foreground app state or accessibility state may have changed. Re-open the app or retry after state is stable.
+- On macOS, use `snapshot --raw --platform macos` to distinguish collector filtering from truly missing AX content.
 
 ## Stop Conditions