@swmansion/argent 0.12.1 → 0.13.1-next.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +12 -11
- package/assets/queries/cpu-hotspots.sql +22 -5
- package/bin/argent-android-devtools-0.1.0.apk +0 -0
- package/bin/darwin/ax-service +0 -0
- package/bin/darwin/simulator-server +0 -0
- package/bin/darwin/tvos-ax-service +0 -0
- package/bin/darwin/tvos-hid-daemon +0 -0
- package/bin/linux/simulator-server +0 -0
- package/bin/linux-arm64/simulator-server +0 -0
- package/dist/cli-cmds.mjs +1945 -1667
- package/dist/installer.mjs +2301 -2038
- package/dist/mcp-server.mjs +237 -464
- package/dist/preview-ui/index.html +117 -58
- package/dist/preview-ui/theme.css +14 -1
- package/dist/preview-window/main.cjs +14 -1
- package/dist/tool-server.cjs +56397 -31889
- package/dylibs/libArgentInjectionBootstrap.dylib +0 -0
- package/dylibs/libKeyboardPatch.dylib +0 -0
- package/dylibs/libNativeDevtoolsIos.dylib +0 -0
- package/dylibs/tvos/libArgentInjectionBootstrap.dylib +0 -0
- package/dylibs/tvos/libKeyboardPatch.dylib +0 -0
- package/dylibs/tvos/libNativeDevtoolsIos.dylib +0 -0
- package/package.json +9 -7
- package/rules/argent.md +10 -5
- package/skills/argent-android-emulator-setup/SKILL.md +1 -0
- package/skills/argent-create-flow/SKILL.md +16 -1
- package/skills/argent-device-interact/SKILL.md +62 -31
- package/skills/argent-native-profiler/SKILL.md +1 -1
- package/skills/argent-test-ui-flow/SKILL.md +14 -5
- package/skills/argent-tv-interact/SKILL.md +60 -0
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@swmansion/argent",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.1-next.0",
|
|
4
4
|
"description": "MCP server for iOS Simulator and Android Emulator control",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"repository": {
|
|
@@ -41,12 +41,14 @@
|
|
|
41
41
|
"scripts/postinstall.cjs"
|
|
42
42
|
],
|
|
43
43
|
"dependencies": {
|
|
44
|
+
"@fails-components/webtransport": "^1.6.3",
|
|
45
|
+
"@fails-components/webtransport-transport-http3-quiche": "^1.6.3",
|
|
44
46
|
"@modelcontextprotocol/sdk": "^1.20.0",
|
|
45
47
|
"tree-sitter": "^0.21.1",
|
|
46
48
|
"tree-sitter-typescript": "^0.23.2"
|
|
47
49
|
},
|
|
48
50
|
"optionalDependencies": {
|
|
49
|
-
"electron": "^42.
|
|
51
|
+
"electron": "^42.5.0"
|
|
50
52
|
},
|
|
51
53
|
"devDependencies": {
|
|
52
54
|
"@argent/cli": "file:../argent-cli",
|
|
@@ -54,14 +56,14 @@
|
|
|
54
56
|
"@argent/mcp": "file:../argent-mcp",
|
|
55
57
|
"@argent/telemetry": "file:../telemetry",
|
|
56
58
|
"@argent/tools-client": "file:../argent-tools-client",
|
|
57
|
-
"@clack/prompts": "^1.
|
|
58
|
-
"@types/node": "^
|
|
59
|
+
"@clack/prompts": "^1.6.0",
|
|
60
|
+
"@types/node": "^26.0.1",
|
|
59
61
|
"@types/semver": "^7.7.1",
|
|
60
62
|
"esbuild": "^0.28.1",
|
|
61
63
|
"picocolors": "^1.1.1",
|
|
62
|
-
"posthog-node": "5.
|
|
63
|
-
"semver": "^7.8.
|
|
64
|
-
"smol-toml": "^1.
|
|
64
|
+
"posthog-node": "5.38.6",
|
|
65
|
+
"semver": "^7.8.5",
|
|
66
|
+
"smol-toml": "^1.7.0",
|
|
65
67
|
"typescript": "^6.0.3",
|
|
66
68
|
"vitest": "^4.1.9",
|
|
67
69
|
"yaml": "^2.8.3"
|
package/rules/argent.md
CHANGED
|
@@ -4,7 +4,7 @@ alwaysApply: true
|
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
<description>
|
|
7
|
-
If argent is installed and configured in this environment, its MCP tools are the preferred form of interaction with the application for iOS simulator, Android emulator,
|
|
7
|
+
If argent is installed and configured in this environment, its MCP tools are the preferred form of interaction with the application for iOS simulator, Android emulator, Chromium (CDP) app, and Vega (Amazon Fire TV) device control; otherwise see `<availability_check>` below before attempting any argent workflow. A "Chromium (CDP) app" is any Chromium runtime exposing a Chrome DevTools Protocol endpoint — an Electron app, or any Chromium-family browser (Chrome/Brave/Edge) launched with `--remote-debugging-port`; all are driven through the same tool surface and tagged `platform: "chromium"`. A "Vega device" is a virtual device (VVD) or physical unit — driven by tv-remote (D-pad) and tagged `platform: "vega"`.
|
|
8
8
|
Running MCP server and managing the Argent toolkit utilises `argent` command - if asked use `argent --help` for reference.
|
|
9
9
|
To check current version of MCP server run `argent --version` command.
|
|
10
10
|
|
|
@@ -15,7 +15,7 @@ Use cases:
|
|
|
15
15
|
- Any tapping, swiping, typing, screenshotting, or inspecting a running app
|
|
16
16
|
- Any code change that affects visible mobile UI, layout, styling, copy, navigation, or screen composition
|
|
17
17
|
- Any request to execute manual QA, UI QA, or visual behavior validation for a mobile app
|
|
18
|
-
- Running, debugging, or testing a React Native app (iOS or
|
|
18
|
+
- Running, debugging, or testing a React Native app (iOS, Android or Vega)
|
|
19
19
|
- Profiling performance or diagnosing re-renders in a React Native app (iOS or Android)
|
|
20
20
|
- Running, debugging, or testing a Chromium (CDP) app — an Electron app (boot with `boot-device` + `electronAppPath`) or a Chromium browser exposing CDP (auto-discovered on port `9222` / `ARGENT_CHROMIUM_PORTS`); on Chromium scroll with `gesture-scroll` and drag with `gesture-drag` — `gesture-swipe` is touch-only
|
|
21
21
|
</description>
|
|
@@ -80,7 +80,7 @@ Decision order:
|
|
|
80
80
|
- When the session ends or the user says they are done: call `stop-all-simulator-servers`.
|
|
81
81
|
If the user started Metro separately, ask whether to call `stop-metro` (specify the port if not 8081).
|
|
82
82
|
- If tools provided by mcp-server are not sufficient and action can be done using `xcrun`, `adb`, or other commands, use the command. Examples: changing device options, performing a device action such as lock, shake, etc.
|
|
83
|
-
- When waiting for an action, do not call `screenshot` repeatedly without a proper wait mechanism.
|
|
83
|
+
- When waiting for an action, do not call `screenshot` repeatedly without a proper wait mechanism. Use the `await-ui-element` tool to block until the UI settles (e.g. wait for an element to become `visible`/`hidden`, or to contain expected `text`) instead of polling.
|
|
84
84
|
</general_rules>
|
|
85
85
|
|
|
86
86
|
<react_native_detection>
|
|
@@ -98,7 +98,7 @@ Load the matching skill before starting work and executing tools from argent-mcp
|
|
|
98
98
|
procedure and edge-case handling for each workflow.
|
|
99
99
|
|
|
100
100
|
PLATFORM DETECTION
|
|
101
|
-
If the user did not specify a platform, call `list-devices` first and pick the booted target — do not default to iOS.
|
|
101
|
+
If the user did not specify a platform, call `list-devices` first and pick the booted target — do not default to iOS. Vega (Amazon Fire TV) devices appear as `platform:"vega"`, when present load `argent-tv-interact`
|
|
102
102
|
|
|
103
103
|
iOS SIMULATOR SETUP
|
|
104
104
|
Skill: `argent-ios-simulator-setup`
|
|
@@ -110,7 +110,12 @@ When: Beginning a task that involves the Android emulator, no emulator running y
|
|
|
110
110
|
|
|
111
111
|
TAPPING, SWIPING, TYPING, GESTURES, SCREENSHOTS, SCROLLING
|
|
112
112
|
Skill: `argent-device-interact`
|
|
113
|
-
When: Performing touch interactions, typing, pressing hardware buttons, launching/restarting apps, opening URLs, rotating device, taking standalone screenshots, or verifying a visible UI code change.
|
|
113
|
+
When: Performing touch interactions, typing, pressing hardware buttons, launching/restarting apps, opening URLs, rotating device, taking standalone screenshots, or verifying a visible UI code change. Phone/tablet iOS and Android only — for any TV target use the TV skill below.
|
|
114
|
+
|
|
115
|
+
TV INTERACTION (APPLE TV / ANDROID TV / FIRE TV)
|
|
116
|
+
Skill: `argent-tv-interact`
|
|
117
|
+
When: Any TV target — a `list-devices` entry with `runtimeKind: "tv"` (Apple TV simulator or Android TV emulator) or `platform:"vega"` / `kind:"vvd"` (Amazon Fire TV / VVD), or the user mentions Apple TV / tvOS / Android TV / leanback / Vega / Fire TV. A TV UI is focus-driven, not touch-driven: drive it with `describe` (read focus) + `tv-remote` (D-pad presses) + `keyboard` (type); `gesture-*` tools do NOT apply. Covers booting the target, app lifecycle, focus navigation, typing, screenshots, and (Vega) VVD lifecycle + Fast Refresh.
|
|
118
|
+
Prompt keywords: apple tv, tvos, android tv, leanback, vega, fire tv, vvd, d-pad
|
|
114
119
|
|
|
115
120
|
SCREENSHOT DIFF & VISUAL REGRESSION
|
|
116
121
|
Skill: `argent-screenshot-diff`
|
|
@@ -23,6 +23,7 @@ Pass the Android serial as `udid` to the unified interaction tools — `gesture-
|
|
|
23
23
|
|
|
24
24
|
## 4. Notes
|
|
25
25
|
|
|
26
|
+
- **Android TV / leanback AVDs** boot through this exact flow (same `boot-device` + `avdName`), but they are **focus-driven, not touch-driven** — do not use `gesture-tap`/`gesture-swipe` on them. `list-devices` tags a leanback device with `runtimeKind: "tv"` (detected via the system feature list, not the serial — a TV AVD's serial looks just like a phone's). When you see `runtimeKind: "tv"`, drive it with the focus-driven tools (`describe` / `tv-remote` / `keyboard`) and the `argent-tv-interact` skill (it covers Android TV as well as Apple TV, including the full TV setup flow).
|
|
26
27
|
- Serials are the adb device id. iOS UDIDs and Android serials are not interchangeable, but you do NOT need to tell the tools which platform — dispatch is automatic.
|
|
27
28
|
- `describe` on Android returns a shallower tree than iOS (no accessibility-service equivalent), but covers most tap-target discovery.
|
|
28
29
|
- `reinstall-app` on Android always installs with `-g` so first-launch runtime permissions are pre-granted.
|
|
@@ -58,6 +58,13 @@ command: "screenshot"
|
|
|
58
58
|
args: "{\"udid\": \"<UDID>\"}"
|
|
59
59
|
```
|
|
60
60
|
|
|
61
|
+
```
|
|
62
|
+
command: "await-ui-element"
|
|
63
|
+
args: "{\"udid\": \"<UDID>\", \"condition\": \"visible\", \"selector\": {\"text\": \"Continue\"}}"
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Record an `await-ui-element` step to **gate** the next step on a screen transition — it blocks until the element is `visible`/`hidden` (or contains `text`), so the following step runs only once the screen has actually settled. If its condition is not met before the timeout, replay **stops at that step** (the steps after it assume the transition happened). Prefer this over a fixed `delayMs`. See the `await-ui-element` section of `argent-device-interact` for the full condition/selector reference.
|
|
67
|
+
|
|
61
68
|
For tools with no arguments, omit `args` entirely.
|
|
62
69
|
|
|
63
70
|
## 5. Important Rules
|
|
@@ -99,7 +106,7 @@ flow-execute { name: "open-settings", project_root: "/Users/dev/MyApp", prereq
|
|
|
99
106
|
Flow files use YAML. The top-level is an object with `executionPrerequisite` (describes required state) and `steps` (array of actions):
|
|
100
107
|
|
|
101
108
|
- `- echo: <message>` — a label
|
|
102
|
-
- `- tool: <name>` with optional `args:` — a tool call.
|
|
109
|
+
- `- tool: <name>` with optional `args:` — a tool call. A tool step may also carry `delayMs: <ms>` to sleep that long before it runs. (`await-ui-element` is an ordinary tool step; see §4 and §10.5 for when to gate a transition with one.)
|
|
103
110
|
|
|
104
111
|
Example `.yaml` file:
|
|
105
112
|
|
|
@@ -111,6 +118,13 @@ steps:
|
|
|
111
118
|
args:
|
|
112
119
|
udid: ABC
|
|
113
120
|
bundleId: com.apple.Preferences
|
|
121
|
+
- echo: Wait for the Settings list to render
|
|
122
|
+
- tool: await-ui-element
|
|
123
|
+
args:
|
|
124
|
+
udid: ABC
|
|
125
|
+
condition: visible
|
|
126
|
+
selector:
|
|
127
|
+
text: General
|
|
114
128
|
- echo: Tap General
|
|
115
129
|
- tool: gesture-tap
|
|
116
130
|
args:
|
|
@@ -207,6 +221,7 @@ After applying a correction, re-run `flow-execute` to verify.
|
|
|
207
221
|
Apply these when recording new flows to reduce future breakage:
|
|
208
222
|
|
|
209
223
|
- **Echo expected state, not just actions.** Write `"On Settings > General screen, about to tap About"` not `"Tap About"`. During diagnosis these tell you what the screen _should_ look like.
|
|
224
|
+
- **Gate transitions with `await-ui-element`, not fixed delays.** After a tap that triggers a navigation, record an `await-ui-element` step that waits for the next screen's element to be `visible` (or a spinner to be `hidden`) before the following step. This removes the **Timing** failure mode in §10.2 (the element is in the tree but the tap fired before the screen settled) and is more reliable than `delayMs` or an extra `screenshot`. An unmet wait stops replay at that step, so a mistimed step can never run blind.
|
|
210
225
|
- **Add screenshot steps after critical navigation.** Insert `screenshot` steps after screen transitions. These produce images in the flow result you can inspect during diagnosis.
|
|
211
226
|
- **Write specific executionPrerequisites.** `"App on home tab, user logged in, simulator UDID is <X>"` — not `"App running"`. Verify with `screenshot` + `describe` before acknowledging.
|
|
212
227
|
- **Prefer launch-app / open-url over navigation chains.** Deep links are more resilient to layout changes than tap sequences.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: argent-device-interact
|
|
3
|
-
description: Interact with an iOS simulator, Android emulator, or Chromium (CDP) app using argent MCP tools. Use when tapping UI elements, performing gestures, scrolling/swiping, typing text, pressing hardware buttons, launching apps, opening URLs, taking screenshots, or checking visible app state after interactions.
|
|
3
|
+
description: Interact with an iOS simulator, Android emulator, or Chromium (CDP) app using argent MCP tools. Use when tapping UI elements, performing gestures, scrolling/swiping, typing text, pressing hardware buttons, launching apps, opening URLs, taking screenshots, waiting for an element to appear or disappear, or checking visible app state after interactions.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
## Unified tool surface
|
|
@@ -13,6 +13,8 @@ All interaction tools below accept a `udid` parameter and auto-dispatch iOS vs A
|
|
|
13
13
|
|
|
14
14
|
**Cookies & storage (Chromium only):** `chromium-cookies` reads/writes cookies via the Network domain (so HttpOnly cookies are visible): `action=get` (optionally scoped by `url`), `set` (`name`, `value`, + `url`/`domain`, optional `secure`/`httpOnly`/`sameSite`/`expires`), `delete` (`name`), `clear` (all). `chromium-storage` reads/writes Web Storage for the active page: `store=local|session`, `action=get` (one `key` or all entries), `set`, `remove`, `clear`. Both are per-origin / active-tab. Handy for seeding auth before a flow or asserting app state after one.
|
|
15
15
|
|
|
16
|
+
> **TV targets (Apple TV / Android TV) are not covered by this skill.** A TV target is **focus-driven, not touch-driven** — the `gesture-*` tools are the wrong tools for it. This applies to both Apple TV simulators (UUID-shaped, identical to iOS) and Android TV / leanback devices (serial-shaped, identical to a phone emulator). If `list-devices` tags your target `runtimeKind: "tv"`, stop and use the `argent-tv-interact` skill: `describe` to read focus, `tv-remote` for remote / D-pad presses, and `keyboard` to type.
|
|
17
|
+
|
|
16
18
|
For platform-specific caveats (Metro `adb reverse`, locked-screen describe errors, etc.), see § 9 Platform-specific notes at the bottom.
|
|
17
19
|
|
|
18
20
|
## 1. Before You Start
|
|
@@ -28,7 +30,7 @@ Use `list-devices` to get a target id. Results are tagged with `platform` (`ios`
|
|
|
28
30
|
1. **Always refer to tapping_rule** from your argent.md rule before tapping.
|
|
29
31
|
2. Before performing interactions, consider whether they can be **dispatched sequentially** - more on that in `run-sequence`.
|
|
30
32
|
3. **Use `gesture-swipe` for lists/scrolling**, not `gesture-custom`, unless you need non-linear movement. On Chromium use `gesture-scroll` instead — `gesture-swipe` is touch-only. Consider whether you need multiple swipes, if yes - use `run-sequence`.
|
|
31
|
-
4. **Tap a text field before typing
|
|
33
|
+
4. **Tap a text field before typing**, then use `keyboard` to enter text.
|
|
32
34
|
5. **Coordinates are normalized** — always 0.0–1.0, not pixels.
|
|
33
35
|
6. **For app navigation, prefer `describe` first.** It works on any screen without app restart. Do not navigate from screenshots on regular in-app screens unless `describe` failed to expose a reliable target. Use `native-describe-screen` only when you need app-scoped UIKit properties.
|
|
34
36
|
|
|
@@ -54,25 +56,25 @@ Common schemes: `messages://`, `settings://`, `maps://?q=<query>`, `tel://<numbe
|
|
|
54
56
|
|
|
55
57
|
## 4. Choosing the Right Tool
|
|
56
58
|
|
|
57
|
-
| Action | Tool
|
|
58
|
-
| ----------------- |
|
|
59
|
-
| Multiple actions | `run-sequence`
|
|
60
|
-
| Open an app | `launch-app`
|
|
61
|
-
| Restart an app | `restart-app`
|
|
62
|
-
| Open URL/scheme | `open-url`
|
|
63
|
-
| Single tap | `gesture-tap`
|
|
64
|
-
| Scroll/swipe | `gesture-swipe`
|
|
65
|
-
| Scroll (Chromium) | `gesture-scroll`
|
|
66
|
-
| Drag (Chromium) | `gesture-drag`
|
|
67
|
-
| Long press | `gesture-custom`
|
|
68
|
-
| Drag & drop | `gesture-custom`
|
|
69
|
-
| Pinch/zoom | `gesture-pinch`
|
|
70
|
-
| Rotation | `gesture-rotate`
|
|
71
|
-
| Custom gesture | `gesture-custom`
|
|
72
|
-
| Hardware key | `button`
|
|
73
|
-
| Type text
|
|
74
|
-
|
|
|
75
|
-
|
|
|
59
|
+
| Action | Tool | Notes |
|
|
60
|
+
| ----------------- | ------------------ | ---------------------------------------------------------------- |
|
|
61
|
+
| Multiple actions | `run-sequence` | Batch steps in one call (no intermediate screenshots) |
|
|
62
|
+
| Open an app | `launch-app` | **Always — never tap home-screen icons** |
|
|
63
|
+
| Restart an app | `restart-app` | Terminate and relaunch by bundle ID |
|
|
64
|
+
| Open URL/scheme | `open-url` | Web pages, deep links, URL schemes |
|
|
65
|
+
| Single tap | `gesture-tap` | Buttons, links, checkboxes |
|
|
66
|
+
| Scroll/swipe | `gesture-swipe` | Straight-line scroll or swipe |
|
|
67
|
+
| Scroll (Chromium) | `gesture-scroll` | Wheel-based; deltas are window fractions, positive deltaY = down |
|
|
68
|
+
| Drag (Chromium) | `gesture-drag` | Sliders, drag-and-drop, text selection |
|
|
69
|
+
| Long press | `gesture-custom` | Context menus, drag start |
|
|
70
|
+
| Drag & drop | `gesture-custom` | Complex drag interactions |
|
|
71
|
+
| Pinch/zoom | `gesture-pinch` | Two-finger pinch with auto-interpolation |
|
|
72
|
+
| Rotation | `gesture-rotate` | Two-finger rotation with auto-interpolation |
|
|
73
|
+
| Custom gesture | `gesture-custom` | Arbitrary touch sequences, optional interpolation |
|
|
74
|
+
| Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton |
|
|
75
|
+
| Type text | `keyboard` | iOS+Android. Supports Enter, Escape, arrows |
|
|
76
|
+
| Rotate device | `rotate` | Orientation changes |
|
|
77
|
+
| Wait for UI | `await-ui-element` | Block until an element is visible/hidden/exists/contains text |
|
|
76
78
|
|
|
77
79
|
## 5. Finding Tap Targets
|
|
78
80
|
|
|
@@ -161,14 +163,6 @@ For long-press, drag-and-drop, and other complex sequences, see `references/gest
|
|
|
161
163
|
|
|
162
164
|
Values: `home`, `back`, `power`, `volumeUp`, `volumeDown`, `appSwitch`, `actionButton`
|
|
163
165
|
|
|
164
|
-
### paste — Type text into focused field (iOS only)
|
|
165
|
-
|
|
166
|
-
```json
|
|
167
|
-
{ "udid": "<UDID>", "text": "Hello, world!" }
|
|
168
|
-
```
|
|
169
|
-
|
|
170
|
-
Tap the field first, then paste. Fall back to `keyboard` if it doesn't work. On Android the call is rejected by the capability gate ("Tool 'paste' is not supported on android") — use `keyboard` directly.
|
|
171
|
-
|
|
172
166
|
### keyboard — Type text or press special keys
|
|
173
167
|
|
|
174
168
|
```json
|
|
@@ -185,6 +179,23 @@ Special keys: `enter`, `escape`, `backspace`, `tab`, `space`, `arrow-up`, `arrow
|
|
|
185
179
|
|
|
186
180
|
Values: `Portrait`, `LandscapeLeft`, `LandscapeRight`, `PortraitUpsideDown`
|
|
187
181
|
|
|
182
|
+
### await-ui-element — Block until a UI element reaches a state
|
|
183
|
+
|
|
184
|
+
Instead of polling `screenshot`/`describe` in a loop, use `await-ui-element` to block server-side until an element reaches an expected state (or `timeoutMs`, default 5000ms, elapses). It polls the same accessibility/DOM tree as `describe`. (For a plain pause, use your own harness sleep — this tool deliberately has no bare-timer mode.)
|
|
185
|
+
|
|
186
|
+
```json
|
|
187
|
+
{ "udid": "<UDID>", "condition": "visible", "selector": { "text": "Continue" } }
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
- `condition`: `exists`, `visible`, `hidden`, or `text`.
|
|
191
|
+
- `selector`: `{ text?, identifier?, role? }` — every provided field must match (case-insensitive substring). `text` matches the element's label or value; `identifier` matches its accessibility id / resource-id / testID; `role` matches its element role (e.g. `AXButton`, `button`, `TextView`, `StaticText`). The synthetic `ROOT` container `describe` prints is never matched, so a `role` like `AXGroup`/`html` won't trivially "match the screen".
|
|
192
|
+
- Prefer a **specific** selector. A loose substring can match several elements, and the tool may then key off one you didn't mean: `text` reads the **first** match in **reading order** (top-to-bottom, left-to-right — the same order `describe` lists them, so it's the one you saw first), while `visible`/`exists` are satisfied by **any** match. Disambiguate with a longer or more exact string, an `identifier`, or a `role` (e.g. pin to a text role like `StaticText` to skip a same-named button). On a `text` timeout the `note` quotes the matched element's text, so you can see which one it landed on.
|
|
193
|
+
- `text` condition also needs `expectedText` (substring the matched element must contain).
|
|
194
|
+
- `hidden` treats a selector that matches **nothing** as already-hidden, so a typo'd selector returns an instant (false) success. Double-check the selector for `hidden` waits — the result `note` flags when the selector never matched any element. (On iOS, if the accessibility backend is down the tree comes back empty; the tool will **not** report `hidden` success off such a degraded read and the `note` surfaces the boot hint instead.)
|
|
195
|
+
- Optional `timeoutMs` (default 5000) and `pollIntervalMs` (default 400).
|
|
196
|
+
|
|
197
|
+
Returns `{ success, elapsed }`; on a timeout `success` is `false` and a `note` explains what was seen.
|
|
198
|
+
|
|
188
199
|
---
|
|
189
200
|
|
|
190
201
|
## 7. Screenshots
|
|
@@ -246,10 +257,12 @@ Use the sequencing when:
|
|
|
246
257
|
|
|
247
258
|
### Allowed tools inside `run-sequence`
|
|
248
259
|
|
|
249
|
-
`gesture-tap`, `gesture-swipe`, `gesture-custom`, `gesture-pinch`, `gesture-rotate`, `button`, `keyboard`, `rotate`
|
|
260
|
+
`gesture-tap`, `gesture-swipe`, `gesture-scroll`, `gesture-drag`, `gesture-custom`, `gesture-pinch`, `gesture-rotate`, `button`, `keyboard`, `rotate`, `await-ui-element`
|
|
250
261
|
|
|
251
262
|
The `udid` is shared — do **not** include it in each step's `args`. Optional `delayMs` per step (default 100ms).
|
|
252
263
|
|
|
264
|
+
Add an `await-ui-element` step to gate a later tap on a screen transition (e.g. tap → wait for the next screen's button → tap it). If its condition is **not** met before the timeout, the sequence stops at that step and the following steps do **not** run — so a mistimed tap can't fire against a screen that never settled.
|
|
265
|
+
|
|
253
266
|
### Examples
|
|
254
267
|
|
|
255
268
|
Scroll down three times:
|
|
@@ -293,7 +306,25 @@ Tap a known button, then scroll down:
|
|
|
293
306
|
}
|
|
294
307
|
```
|
|
295
308
|
|
|
296
|
-
|
|
309
|
+
Tap, wait for the next screen, then act on it — the `await-ui-element` step **gates** the tap after it:
|
|
310
|
+
|
|
311
|
+
```json
|
|
312
|
+
{
|
|
313
|
+
"udid": "<UDID>",
|
|
314
|
+
"steps": [
|
|
315
|
+
{ "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.9 } },
|
|
316
|
+
{
|
|
317
|
+
"tool": "await-ui-element",
|
|
318
|
+
"args": { "condition": "visible", "selector": { "text": "Continue" } }
|
|
319
|
+
},
|
|
320
|
+
{ "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.5 } }
|
|
321
|
+
]
|
|
322
|
+
}
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
Prefer this over a fixed `delayMs` when a step depends on a screen transition: it adapts to real load time, and if the condition is not met before the timeout the sequence **stops there** so the next tap can't fire against a screen that never settled.
|
|
326
|
+
|
|
327
|
+
Stops on the first error (or unmet `await-ui-element` condition) and returns partial results.
|
|
297
328
|
|
|
298
329
|
---
|
|
299
330
|
|
|
@@ -87,7 +87,7 @@ To revisit a previous trace:
|
|
|
87
87
|
Bottlenecks are categorized by severity:
|
|
88
88
|
|
|
89
89
|
- **RED**: CPU functions taking >15% of total time, all UI hangs, and **attributed** memory leaks (those with a resolved responsible frame). These require immediate attention.
|
|
90
|
-
- **YELLOW**: CPU functions taking
|
|
90
|
+
- **YELLOW**: CPU functions taking 3-15% of total time, and **unattributed** memory leaks (`<Call stack limit reached>`, no library — see the memory-leaks caveat below). Worth investigating but may be acceptable.
|
|
91
91
|
|
|
92
92
|
Each bottleneck type indicates a different class of problem:
|
|
93
93
|
|
|
@@ -29,7 +29,7 @@ For implementation tasks that modify visible UI, this workflow can also serve as
|
|
|
29
29
|
- **Permission prompts / system modal overlays**: try `describe` first. Fall back to `screenshot` only if the overlay is not exposed reliably.
|
|
30
30
|
- **Fallback**: use `screenshot` to estimate where the desired component is, then verify immediately after the action.
|
|
31
31
|
3. **Interact**: Perform the action (`gesture-tap`, `gesture-swipe`, `keyboard`, `button`, ...) — you receive a screenshot automatically.
|
|
32
|
-
4. **Verify**: Check the returned screenshot for expected results. If it shows a loading/transitional state,
|
|
32
|
+
4. **Verify**: Check the returned screenshot for expected results. If it shows a loading/transitional state, prefer blocking until it settles with `await-ui-element` (expected element `visible`, or a spinner `hidden`) over a guessed delay — but only with a selector you can trust (`text`/`identifier`/`role`) that the screen is known to have or that you saw in a prior `describe`; a guessed one just times out. Otherwise use a short fixed wait. Pick evidence by what's being asserted:
|
|
33
33
|
- **Visual** (layout, spacing, color, typography, image/icon rendering, clipping, overflow, text rendering): prefer `screenshot-diff` against the baseline captured in step 1 — it surfaces pixel-visible changes the auto-screenshot might miss. Fall back to visual inspection of the auto-screenshot only when a stable baseline isn't available.
|
|
34
34
|
- **Structural** (navigation state, element existence, accessibility labels/values, selection, hierarchy, route): verify with `describe`, `debugger-component-tree`, or `native-describe-screen`.
|
|
35
35
|
- **Runtime / log / network** (console errors, API calls, persistence, timing): verify with `view-network-logs`, `debugger-log-registry`, `debugger-evaluate`, or targeted tests.
|
|
@@ -58,9 +58,9 @@ Steps:
|
|
|
58
58
|
```
|
|
59
59
|
1. screenshot → see login screen
|
|
60
60
|
2. gesture-tap { x: 0.5, y: 0.4 } → tap email field
|
|
61
|
-
3.
|
|
61
|
+
3. keyboard { text: "user@example.com" }
|
|
62
62
|
4. gesture-tap { x: 0.5, y: 0.55 } → tap password field
|
|
63
|
-
5.
|
|
63
|
+
5. keyboard { text: "password123" }
|
|
64
64
|
6. gesture-tap { x: 0.5, y: 0.7 } → tap Login button
|
|
65
65
|
7. screenshot → verify home screen appeared
|
|
66
66
|
```
|
|
@@ -89,11 +89,20 @@ Steps:
|
|
|
89
89
|
8. Report combined verdict from expected behavior, visual inspection, diff summary, and structural evidence.
|
|
90
90
|
```
|
|
91
91
|
|
|
92
|
+
### Wait for a loading spinner
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
1. gesture-tap { x: 0.5, y: 0.7 } → trigger an action that fetches data
|
|
96
|
+
2. screenshot → loading spinner is showing
|
|
97
|
+
3. await-ui-element { condition: hidden, selector: { text: "Loading" } } → block until the fetch finishes and the spinner disappears
|
|
98
|
+
4. describe / screenshot → verify the fetched content rendered
|
|
99
|
+
```
|
|
100
|
+
|
|
92
101
|
---
|
|
93
102
|
|
|
94
103
|
## 4. Recovery Pattern
|
|
95
104
|
|
|
96
|
-
- If
|
|
105
|
+
- If a screen is mid-transition or loading: block until it settles with `await-ui-element` (wait for the target element to be `visible`, or the spinner/placeholder to be `hidden`) instead of a blind fixed delay, then re-check. Fall back to a fixed wait + `screenshot` only when no element reliably marks the transition.
|
|
97
106
|
- If tap misses target: re-run discovery tool (`describe` / `debugger-component-tree`), retry once with new coordinates.
|
|
98
107
|
- If a permission dialog or modal is visible: re-run `describe` first. Stay in screenshot-driven navigation only when the overlay is not exposed reliably, then switch back to `describe` / `debugger-component-tree` as soon as it is dismissed.
|
|
99
108
|
- If tap fails twice at same coordinates: stop, re-discover, report if element not found.
|
|
@@ -101,7 +110,7 @@ Steps:
|
|
|
101
110
|
|
|
102
111
|
## Tips
|
|
103
112
|
|
|
104
|
-
- **
|
|
113
|
+
- **Wait on the UI, don't poll.** When a step needs the screen to change first, gate it with `await-ui-element` (block until an element is `visible`/`hidden` or contains `text`) rather than repeated `screenshot` calls with fixed sleeps. See the `await-ui-element` section of `argent-device-interact`.
|
|
105
114
|
- **Use `gesture-custom` for long-press** context menus (800ms hold).
|
|
106
115
|
- **Report clearly**: state what you expected, what you saw, and the verdict.
|
|
107
116
|
- **Permission modals**: try `describe` first. Use `screenshot` only as fallback, tap one visible button at a time, and verify with the returned screenshot before continuing.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: argent-tv-interact
|
|
3
|
+
description: Control and inspect TV apps via argent — Apple TV (tvOS), Android TV (leanback), and Amazon Fire TV (Vega). Boot the target, read focus, navigate with the D-pad remote, type, and screenshot. Use when a task targets a TV (runtimeKind "tv", or platform "vega"), or mentions Apple TV / tvOS / Android TV / leanback / Vega / Fire TV / VVD.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Argent TV (Apple TV + Android TV + Fire TV)
|
|
7
|
+
|
|
8
|
+
## Critical
|
|
9
|
+
|
|
10
|
+
- A TV is **focus-driven, not touch-driven.** Drive every interaction with `describe` + `tv-remote` + `keyboard`; never use `gesture-*` / coordinate taps — they don't apply on any TV platform.
|
|
11
|
+
- **Always `describe` before navigating** to find the live cursor and your target — never guess focus from a screenshot. The cursor is the focused element; on **Vega** the toolkit often leaves `focused` false and marks the highlighted item `[selected]`, so treat `[selected]` as the cursor when nothing reports `[focused]`.
|
|
12
|
+
- Pass the `udid` from `list-devices` — an Apple TV simulator UDID or an Android TV / Vega `serial`. Dispatch is automatic from the id; the same tools drive all three.
|
|
13
|
+
|
|
14
|
+
## The navigation loop
|
|
15
|
+
|
|
16
|
+
1. `describe` — find the cursor and your target (returns the focused element + all focusable ones, not a tap tree).
|
|
17
|
+
2. `tv-remote` — move focus toward the target. Prefer **one** call with a path ending in `select`, e.g. `{button:["down","right","select"]}`; count rows/columns from the frames to build the path.
|
|
18
|
+
3. `describe` again to confirm. On a miss, repeat.
|
|
19
|
+
|
|
20
|
+
## Tools
|
|
21
|
+
|
|
22
|
+
- `describe {udid}` — focus view: the focused / `[selected]` element + focusable elements with labels and normalized frames. The discovery tool — call before and after navigating. Empty tree → see the per-platform notes.
|
|
23
|
+
- `tv-remote {udid, button}` — D-pad / remote. `button` is one key **or a whole path** (run in one call). Keys: `up`/`down`/`left`/`right`, `select`, `back`, `menu`, `home`, `playPause`, plus media keys `rewind`/`fastForward`/`next`/`previous`/`volumeUp`/`volumeDown`/`mute`. Single: `{button:"down"}`; repeat: `{button:"down", repeat:3}`; path: `{button:["up","right","select"]}`.
|
|
24
|
+
- `keyboard {udid, text}` — type into the focused field (focus it with `tv-remote` first). Named `key` presses (e.g. `{key:"enter"}`) work on Vega; on Apple TV / Android TV move focus with `tv-remote` instead.
|
|
25
|
+
- `launch-app` / `restart-app` / `reinstall-app {udid, bundleId}` — `bundleId` from the app manifest. Vega `reinstall-app` takes `appPath` = a `.vpkg`.
|
|
26
|
+
- `screenshot {udid, scale?}` — Apple TV via `xcrun simctl io` (downscaled); Android TV / Vega host-side via `adb` / `screencap`.
|
|
27
|
+
|
|
28
|
+
## Per-platform
|
|
29
|
+
|
|
30
|
+
### Apple TV (tvOS simulator)
|
|
31
|
+
|
|
32
|
+
- Boot like any iOS sim (`boot-device`); the AX + HID daemons auto-start on the first `describe` / `tv-remote` (first call may take a few seconds). Give the RN bundle a few seconds to render before the first `describe`.
|
|
33
|
+
- Media-transport / volume keys are **rejected** — the sim's HID stack ignores them (they work on Android TV / Vega).
|
|
34
|
+
- Dev build: `open-url {udid, url:"<scheme>://expo-development-client/?url=http%3A%2F%2F<HOST_IP>%3A8081"}` (`<HOST_IP>` = your Mac's LAN IP, shown on the launcher).
|
|
35
|
+
|
|
36
|
+
### Android TV (leanback emulator)
|
|
37
|
+
|
|
38
|
+
- Boot the leanback AVD like any emulator — see `argent-android-emulator-setup`.
|
|
39
|
+
- **`describe` may report zero focusables on a screen with visible tiles**: many `react-native-tvos` screens use RN's own focus engine, invisible to the OS accessibility tree. `describe` auto-falls-back to the full UI tree (and says so in the hint); `tv-remote` still moves focus, so drive blind + `screenshot` to confirm.
|
|
40
|
+
- Dev build: `adb -s <serial> reverse tcp:8081 tcp:8081`, deep-link `<pkg>://expo-development-client/?url=http%3A%2F%2F10.0.2.2%3A8081`, dismiss the first dev-menu with `adb shell input keyevent KEYCODE_DPAD_CENTER` (not Back — Back exits the app).
|
|
41
|
+
|
|
42
|
+
### Fire TV (Vega / VVD)
|
|
43
|
+
|
|
44
|
+
- `list-devices` shows a `serial` (use as `udid`) and a `vvdImage`. `boot-device {vvdImage}` (e.g. `"tv"`) starts the single SDK-managed VVD; skip if one already runs.
|
|
45
|
+
- **Stop the VVD** with `vega virtual-device stop` in your shell. The CLI only tracks VVDs it started in the foreground, so it may report "not running" for one started via `boot-device`; to restart that one use `boot-device {vvdImage, force:true}` (stops then re-boots).
|
|
46
|
+
- Empty `describe` tree → `restart-app` (the automation toolkit attaches at launch), then retry. Input ignored → enable developer mode in the VVD: `vsm developer-mode enable`.
|
|
47
|
+
- Editing `node_modules` has no effect on a Release build — only Debug `.vpkg` builds load patchable JS.
|
|
48
|
+
- Profiling / crashes → `amazon-devices-buildertools-mcp` server (`analyze_perfetto_traces`, `get_app_hot_functions`, `symbolicate_acr`); docs via its `search_documentation` tool.
|
|
49
|
+
|
|
50
|
+
## Common gotchas
|
|
51
|
+
|
|
52
|
+
- **Empty focus right after `launch-app` / `restart-app`** is the splash / loading window — `describe` retries internally; wait ~2-3s and retry on a cold start.
|
|
53
|
+
- Passing a phone/tablet (`runtimeKind: "mobile"`) udid to `tv-remote` fails with a clear "tvOS-only" / "Android-TV-only" error — pick a TV target from `list-devices`.
|
|
54
|
+
|
|
55
|
+
## Fast Refresh (dev builds)
|
|
56
|
+
|
|
57
|
+
Needs a Debug build + Metro running. argent only _connects_ to Metro — start Metro and port-forward yourself (any platform). Metro is fixed on **:8081**.
|
|
58
|
+
|
|
59
|
+
- **Apple TV / Android TV:** use the dev-build deep-links above; `npm start` for Metro.
|
|
60
|
+
- **Vega:** build/install a Debug `.vpkg` (`vega device install-app -p <path>`), `npm start`, `vega device start-port-forwarding --port 8081 --forward false`, then `vega device launch-app -a <appId>`. Confirm `http://localhost:8081/json/list` shows a `Hermes React Native` target; `.tsx` edits then hot-reload.
|