agent-device 0.1.4 → 0.1.5

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-device",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "description": "Unified control plane for physical and virtual devices via an agent-driven CLI.",
   "license": "MIT",
   "author": "Callstack",
@@ -37,6 +37,7 @@
     "!ios-runner/**/.swiftpm",
     "!ios-runner/**/xcuserdata",
     "!ios-runner/**/*.xcuserstate",
+    "skills",
     "src",
     "README.md",
     "LICENSE"

package/skills/agent-device/SKILL.md

@@ -0,0 +1,156 @@
+---
+name: agent-device
+description: Automates mobile and simulator interactions for iOS and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info on mobile devices or simulators.
+---
+
+# Mobile Automation with agent-device
+
+## Quick start
+
+```bash
+agent-device open Settings --platform ios
+agent-device snapshot -i
+agent-device snapshot -s @e3
+agent-device click @e3
+agent-device wait text "Camera"
+agent-device alert wait 10000
+agent-device fill @e5 "test"
+agent-device close
+```
+
+## Core workflow
+
+1. Open app or just boot device: `open [app]`
+2. Snapshot: `snapshot -i` to get compact refs
+3. Interact using refs (`click @eN`, `fill @eN "text"`)
+4. Re-snapshot after navigation or UI changes
+5. Close session when done
+
+## Commands
+
+### Navigation
+
+```bash
+agent-device open [app]           # Boot device/simulator; optionally launch app
+agent-device close [app]          # Close app or just end session
+agent-device session list         # List active sessions
+```
+
+### Snapshot (page analysis)
+
+```bash
+agent-device snapshot                 # Full accessibility tree
+agent-device snapshot -i              # Interactive elements only (recommended)
+agent-device snapshot -c              # Compact output
+agent-device snapshot -d 3            # Limit depth
+agent-device snapshot -s "Camera"      # Scope to label/identifier
+agent-device snapshot --raw           # Raw node output
+agent-device snapshot --backend hybrid # Default: best speed vs correctness trade-off (AX fast, XCTest complete)
+agent-device snapshot --backend ax    # macOS Accessibility tree (fast, needs permissions)
+agent-device snapshot --backend xctest # XCTest snapshot (slow, no permissions)
+```
+
+Hybrid will automatically fill empty containers (e.g. `group`, `tab bar`) by scoping XCTest to the container label.
+It is recommended because AX is fast but can miss UI details, while XCTest is slower but more complete.
+If you want explicit control or AX is unavailable, use `--backend xctest`.
+In practice, if AX returns a `Tab Bar` group with no children, hybrid will run a scoped XCTest snapshot for `Tab Bar` and insert those nodes under the group.
+
+### Find (semantic)
+
+```bash
+agent-device find "Sign In" click
+agent-device find text "Sign In" click
+agent-device find label "Email" fill "user@example.com"
+agent-device find value "Search" type "query"
+agent-device find role button click
+agent-device find id "com.example:id/login" click
+agent-device find "Settings" wait 10000
+agent-device find "Settings" exists
+```
+
+### Settings helpers (simulators)
+
+```bash
+agent-device settings wifi on
+agent-device settings wifi off
+agent-device settings airplane on
+agent-device settings airplane off
+agent-device settings location on
+agent-device settings location off
+```
+
+Note: iOS wifi/airplane toggles status bar indicators, not actual network state.
+Airplane off clears status bar overrides.
+
+### App state
+
+```bash
+agent-device appstate
+agent-device apps --metadata --platform ios
+agent-device apps --metadata --platform android
+```
+
+### Interactions (use @refs from snapshot)
+
+```bash
+agent-device click @e1
+agent-device focus @e2
+agent-device fill @e2 "text"           # Tap then type
+agent-device type "text"               # Type into focused field
+agent-device press 300 500             # Tap by coordinates
+agent-device long-press 300 500 800    # Long press (where supported)
+agent-device scroll down 0.5
+agent-device back
+agent-device home
+agent-device app-switcher
+agent-device wait 1000
+agent-device wait text "Settings"
+agent-device alert get
+```
+
+### Get information
+
+```bash
+agent-device get text @e1
+agent-device get attrs @e1
+agent-device screenshot --out out.png
+```
+
+### Trace logs (AX/XCTest)
+
+```bash
+agent-device trace start               # Start trace capture
+agent-device trace start ./trace.log   # Start trace capture to path
+agent-device trace stop                # Stop trace capture
+agent-device trace stop ./trace.log    # Stop and move trace log
+```
+
+### Devices and apps
+
+```bash
+agent-device devices
+agent-device apps --platform ios
+agent-device apps --platform android          # default: launchable only
+agent-device apps --platform android --all
+agent-device apps --platform android --user-installed
+```
+
+## Best practices
+
+- Always snapshot right before interactions; refs invalidate on UI changes.
+- Prefer `snapshot -i` to reduce output size.
+- On iOS, hybrid is the default and uses AX first, so Accessibility permission is still required.
+- If AX returns the Simulator window or empty tree, restart Simulator or use `--backend xctest`.
+- Use `--session <name>` for parallel sessions; avoid device contention.
+
+## References
+
+- [references/snapshot-refs.md](references/snapshot-refs.md)
+- [references/session-management.md](references/session-management.md)
+- [references/permissions.md](references/permissions.md)
+- [references/recording.md](references/recording.md)
+- [references/coordinate-system.md](references/coordinate-system.md)
+
+## Missing features roadmap (high level)
+
+See [references/missing-features.md](references/missing-features.md) for planned parity with agent-browser.

package/skills/agent-device/references/coordinate-system.md

@@ -0,0 +1,8 @@
+# Coordinate System
+
+All coordinate-based actions use device screen coordinates:
+
+- Origin: top-left of the device screen
+- Units: device points for iOS, pixels for Android
+
+Use screenshots to reason about coordinates.

package/skills/agent-device/references/permissions.md

@@ -0,0 +1,20 @@
+# Permissions and Setup
+
+## iOS AX snapshot
+
+Hybrid snapshot (default) is recommended for best speed vs correctness; it uses macOS Accessibility APIs and requires permission:
+
+System Settings > Privacy & Security > Accessibility
+
+If permission is missing, use:
+
+```bash
+agent-device snapshot --backend xctest --platform ios
+```
+
+Hybrid/AX is fast; XCTest is slower but does not require permissions.
+
+## Simulator troubleshooting
+
+- If AX shows the Simulator chrome instead of app, restart Simulator.
+- If AX returns empty, restart Simulator and re-open app.

package/skills/agent-device/references/session-management.md

@@ -0,0 +1,22 @@
+# Session Management
+
+## Named sessions
+
+```bash
+agent-device --session auth open Settings --platform ios
+agent-device --session auth snapshot -i --platform ios
+```
+
+Sessions isolate device context. A device can only be held by one session at a time.
+
+## Best practices
+
+- Name sessions semantically.
+- Close sessions when done.
+- Use separate devices for parallel work.
+
+## Listing sessions
+
+```bash
+agent-device session list
+```

package/skills/agent-device/references/snapshot-refs.md

@@ -0,0 +1,49 @@
+# Snapshot + Refs Workflow (Mobile)
+
+## Purpose
+
+Refs let agents interact without repeating full UI trees. Snapshot -> refs -> click/fill.
+
+## Snapshot
+
+```bash
+agent-device snapshot -i --platform ios
+```
+
+Output:
+
+```
+Page: com.apple.Preferences
+App: com.apple.Preferences
+
+@e1 [ioscontentgroup]
+  @e2 [button] "Camera"
+  @e3 [button] "Privacy & Security"
+```
+
+## Using refs
+
+```bash
+agent-device click @e2 --platform ios
+agent-device fill @e5 "test" --platform ios
+```
+
+## Ref lifecycle
+
+Refs become invalid when UI changes (navigation, modal, dynamic list updates).
+Always re-snapshot after any transition.
+
+## Scope snapshots
+
+Use `-s` to scope to labels/identifiers. This reduces size and speeds up results:
+
+```bash
+agent-device snapshot -i -s "Camera" --platform ios
+agent-device snapshot -i -s @e3 --platform ios
+```
+
+## Troubleshooting
+
+- Ref not found: re-snapshot.
+- AX returns Simulator window: restart Simulator and re-run.
+- AX empty: verify Accessibility permission or use `--backend xctest` (hybrid is recommended because AX is fast but can miss UI details, while XCTest is slower but more complete).

package/skills/agent-device/references/video-recording.md

@@ -0,0 +1,39 @@
+# Video Recording
+
+Capture device automation sessions as video for debugging, documentation, or verification
+
+## iOS Simulator
+
+Use `agent-device record` commands (wrapper around simctl):
+
+```bash
+# Start recording
+agent-device record start ./recordings/ios.mov
+
+# Perform actions
+agent-device open App
+agent-device snapshot
+agent-device click @e3
+agent-device close
+
+# Stop recording
+agent-device record stop
+```
+
+## Android Emulator/Device
+
+Use `agent-device record` commands (wrapper around adb):
+
+```bash
+# Start recording
+agent-device record start ./recordings/android.mp4
+
+# Perform actions
+agent-device open App
+agent-device snapshot
+agent-device click @e3
+agent-device close
+
+# Stop recording
+agent-device record stop
+```