@swmansion/argent 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +13 -8
- package/assets/queries/cpu-hotspots.sql +22 -5
- package/bin/argent-android-devtools-0.1.0.apk +0 -0
- package/bin/darwin/ax-service +0 -0
- package/dist/cli-cmds.mjs +7203 -758
- package/dist/cli.js +3 -0
- package/dist/cli.js.map +1 -1
- package/dist/installer.mjs +9733 -2819
- package/dist/mcp-server.mjs +2751 -116
- package/dist/preview-ui/index.html +7 -17
- package/dist/tool-server.cjs +109698 -64722
- package/dylibs/libArgentInjectionBootstrap.dylib +0 -0
- package/dylibs/libKeyboardPatch.dylib +0 -0
- package/dylibs/libNativeDevtoolsIos.dylib +0 -0
- package/package.json +11 -7
- package/rules/argent.md +24 -4
- package/skills/argent-create-flow/SKILL.md +16 -1
- package/skills/argent-device-interact/SKILL.md +60 -31
- package/skills/argent-native-profiler/SKILL.md +3 -3
- package/skills/argent-test-ui-flow/SKILL.md +14 -5
- package/skills/argent-vega/SKILL.md +76 -0
|
Binary file
|
|
Binary file
|
|
Binary file
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@swmansion/argent",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.0",
|
|
4
4
|
"description": "MCP server for iOS Simulator and Android Emulator control",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"repository": {
|
|
@@ -41,7 +41,9 @@
|
|
|
41
41
|
"scripts/postinstall.cjs"
|
|
42
42
|
],
|
|
43
43
|
"dependencies": {
|
|
44
|
-
"@modelcontextprotocol/sdk": "^1.20.0"
|
|
44
|
+
"@modelcontextprotocol/sdk": "^1.20.0",
|
|
45
|
+
"tree-sitter": "^0.21.1",
|
|
46
|
+
"tree-sitter-typescript": "^0.23.2"
|
|
45
47
|
},
|
|
46
48
|
"optionalDependencies": {
|
|
47
49
|
"electron": "^42.4.1"
|
|
@@ -50,16 +52,18 @@
|
|
|
50
52
|
"@argent/cli": "file:../argent-cli",
|
|
51
53
|
"@argent/installer": "file:../argent-installer",
|
|
52
54
|
"@argent/mcp": "file:../argent-mcp",
|
|
55
|
+
"@argent/telemetry": "file:../telemetry",
|
|
53
56
|
"@argent/tools-client": "file:../argent-tools-client",
|
|
54
|
-
"@clack/prompts": "^1.1
|
|
55
|
-
"@types/node": "^25.9.
|
|
57
|
+
"@clack/prompts": "^1.5.1",
|
|
58
|
+
"@types/node": "^25.9.3",
|
|
56
59
|
"@types/semver": "^7.7.1",
|
|
57
|
-
"esbuild": "^0.28.
|
|
60
|
+
"esbuild": "^0.28.1",
|
|
58
61
|
"picocolors": "^1.1.1",
|
|
59
|
-
"
|
|
62
|
+
"posthog-node": "5.35.0",
|
|
63
|
+
"semver": "^7.8.4",
|
|
60
64
|
"smol-toml": "^1.6.1",
|
|
61
65
|
"typescript": "^6.0.3",
|
|
62
|
-
"vitest": "^4.1.
|
|
66
|
+
"vitest": "^4.1.9",
|
|
63
67
|
"yaml": "^2.8.3"
|
|
64
68
|
}
|
|
65
69
|
}
|
package/rules/argent.md
CHANGED
|
@@ -4,7 +4,7 @@ alwaysApply: true
|
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
<description>
|
|
7
|
-
|
|
7
|
+
If argent is installed and configured in this environment, its MCP tools are the preferred form of interaction with the application for iOS simulator, Android emulator, Chromium (CDP) app, and Vega (Amazon Fire TV) device control; otherwise see `<availability_check>` below before attempting any argent workflow. A "Chromium (CDP) app" is any Chromium runtime exposing a Chrome DevTools Protocol endpoint — an Electron app, or any Chromium-family browser (Chrome/Brave/Edge) launched with `--remote-debugging-port`; all are driven through the same tool surface and tagged `platform: "chromium"`. A "Vega device" is a virtual device (VVD) or physical unit — driven by tv-remote (D-pad) and tagged `platform: "vega"`.
|
|
8
8
|
Running MCP server and managing the Argent toolkit utilises `argent` command - if asked use `argent --help` for reference.
|
|
9
9
|
To check current version of MCP server run `argent --version` command.
|
|
10
10
|
|
|
@@ -15,11 +15,26 @@ Use cases:
|
|
|
15
15
|
- Any tapping, swiping, typing, screenshotting, or inspecting a running app
|
|
16
16
|
- Any code change that affects visible mobile UI, layout, styling, copy, navigation, or screen composition
|
|
17
17
|
- Any request to execute manual QA, UI QA, or visual behavior validation for a mobile app
|
|
18
|
-
- Running, debugging, or testing a React Native app (iOS or
|
|
18
|
+
- Running, debugging, or testing a React Native app (iOS, Android or Vega)
|
|
19
19
|
- Profiling performance or diagnosing re-renders in a React Native app (iOS or Android)
|
|
20
20
|
- Running, debugging, or testing a Chromium (CDP) app — an Electron app (boot with `boot-device` + `electronAppPath`) or a Chromium browser exposing CDP (auto-discovered on port `9222` / `ARGENT_CHROMIUM_PORTS`); on Chromium scroll with `gesture-scroll` and drag with `gesture-drag` — `gesture-swipe` is touch-only
|
|
21
21
|
</description>
|
|
22
22
|
|
|
23
|
+
<availability_check>
|
|
24
|
+
<important>Run this check once per session, before the first argent tool call or `argent` command. Do not re-probe before later calls.</important>
|
|
25
|
+
|
|
26
|
+
Confirm argent is available:
|
|
27
|
+
|
|
28
|
+
1. Are `mcp__argent__*` tools in your tool list? If none are present, argent is not available.
|
|
29
|
+
2. If still unsure, run `command -v argent`. A non-zero exit means the CLI is not on PATH.
|
|
30
|
+
|
|
31
|
+
If argent IS available, ignore the rest of this block and follow this rule normally.
|
|
32
|
+
|
|
33
|
+
If argent is ABSENT, treat it as an expected state, not an error to retry. Do not call `mcp__argent__*` tools, do not run `argent` commands, and do not attempt any argent workflow. Tell the user once, and ask if you should continue without argent:
|
|
34
|
+
|
|
35
|
+
> Argent isn't installed in this environment. To enable the mobile/Chromium tooling this repo is configured for, run `npx @swmansion/argent init -y` (or `npm i -g @swmansion/argent && argent init -y`).
|
|
36
|
+
> </availability_check>
|
|
37
|
+
|
|
23
38
|
<tapping_rule>
|
|
24
39
|
<important>**Never** derive tap coordinates from a screenshot</important>
|
|
25
40
|
Before **every** tap, you MUST call a discovery tool and extract coordinates from the result. This is not optional. Preferred tools are, in order:
|
|
@@ -65,7 +80,7 @@ Decision order:
|
|
|
65
80
|
- When the session ends or the user says they are done: call `stop-all-simulator-servers`.
|
|
66
81
|
If the user started Metro separately, ask whether to call `stop-metro` (specify the port if not 8081).
|
|
67
82
|
- If tools provided by mcp-server are not sufficient and action can be done using `xcrun`, `adb`, or other commands, use the command. Examples: changing device options, performing a device action such as lock, shake, etc.
|
|
68
|
-
- When waiting for an action, do not call `screenshot` repeatedly without a proper wait mechanism.
|
|
83
|
+
- When waiting for an action, do not call `screenshot` repeatedly without a proper wait mechanism. Use the `await-ui-element` tool to block until the UI settles (e.g. wait for an element to become `visible`/`hidden`, or to contain expected `text`) instead of polling.
|
|
69
84
|
</general_rules>
|
|
70
85
|
|
|
71
86
|
<react_native_detection>
|
|
@@ -83,7 +98,7 @@ Load the matching skill before starting work and executing tools from argent-mcp
|
|
|
83
98
|
procedure and edge-case handling for each workflow.
|
|
84
99
|
|
|
85
100
|
PLATFORM DETECTION
|
|
86
|
-
If the user did not specify a platform, call `list-devices` first and pick the booted target — do not default to iOS.
|
|
101
|
+
If the user did not specify a platform, call `list-devices` first and pick the booted target — do not default to iOS. Vega (Amazon Fire TV) devices appear as `platform:"vega"`, when present load `argent-vega`
|
|
87
102
|
|
|
88
103
|
iOS SIMULATOR SETUP
|
|
89
104
|
Skill: `argent-ios-simulator-setup`
|
|
@@ -93,6 +108,11 @@ ANDROID EMULATOR SETUP
|
|
|
93
108
|
Skill: `argent-android-emulator-setup`
|
|
94
109
|
When: Beginning a task that involves the Android emulator, no emulator running yet, need an adb serial, or about to install an APK.
|
|
95
110
|
|
|
111
|
+
VEGA / AMAZON FIRE TV APP CONTROL
|
|
112
|
+
Skill: `argent-vega`
|
|
113
|
+
When: Any task involving a Vega / Amazon Fire TV device (a `platform:"vega"` / `kind:"vvd"` entry in `list-devices`, or the user mentions Vega / Fire TV / VVD). Covers list/launch/restart/reinstall apps, on-screen element discovery via `describe`, D-pad navigation with the `tv-remote` tool (Vega is remote-driven, not touch), typing, screenshots, Fast Refresh setup, and VVD lifecycle (start/stop via the `vega` CLI — argent has no Vega stop tool).
|
|
114
|
+
Prompt keywords: vega, fire tv, vvd, virtual device, d-pad
|
|
115
|
+
|
|
96
116
|
TAPPING, SWIPING, TYPING, GESTURES, SCREENSHOTS, SCROLLING
|
|
97
117
|
Skill: `argent-device-interact`
|
|
98
118
|
When: Performing touch interactions, typing, pressing hardware buttons, launching/restarting apps, opening URLs, rotating device, taking standalone screenshots, or verifying a visible UI code change.
|
|
@@ -58,6 +58,13 @@ command: "screenshot"
|
|
|
58
58
|
args: "{\"udid\": \"<UDID>\"}"
|
|
59
59
|
```
|
|
60
60
|
|
|
61
|
+
```
|
|
62
|
+
command: "await-ui-element"
|
|
63
|
+
args: "{\"udid\": \"<UDID>\", \"condition\": \"visible\", \"selector\": {\"text\": \"Continue\"}}"
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Record an `await-ui-element` step to **gate** the next step on a screen transition — it blocks until the element is `visible`/`hidden` (or contains `text`), so the following step runs only once the screen has actually settled. If its condition is not met before the timeout, replay **stops at that step** (the steps after it assume the transition happened). Prefer this over a fixed `delayMs`. See the `await-ui-element` section of `argent-device-interact` for the full condition/selector reference.
|
|
67
|
+
|
|
61
68
|
For tools with no arguments, omit `args` entirely.
|
|
62
69
|
|
|
63
70
|
## 5. Important Rules
|
|
@@ -99,7 +106,7 @@ flow-execute { name: "open-settings", project_root: "/Users/dev/MyApp", prereq
|
|
|
99
106
|
Flow files use YAML. The top-level is an object with `executionPrerequisite` (describes required state) and `steps` (array of actions):
|
|
100
107
|
|
|
101
108
|
- `- echo: <message>` — a label
|
|
102
|
-
- `- tool: <name>` with optional `args:` — a tool call.
|
|
109
|
+
- `- tool: <name>` with optional `args:` — a tool call. A tool step may also carry `delayMs: <ms>` to sleep that long before it runs. (`await-ui-element` is an ordinary tool step; see §4 and §10.5 for when to gate a transition with one.)
|
|
103
110
|
|
|
104
111
|
Example `.yaml` file:
|
|
105
112
|
|
|
@@ -111,6 +118,13 @@ steps:
|
|
|
111
118
|
args:
|
|
112
119
|
udid: ABC
|
|
113
120
|
bundleId: com.apple.Preferences
|
|
121
|
+
- echo: Wait for the Settings list to render
|
|
122
|
+
- tool: await-ui-element
|
|
123
|
+
args:
|
|
124
|
+
udid: ABC
|
|
125
|
+
condition: visible
|
|
126
|
+
selector:
|
|
127
|
+
text: General
|
|
114
128
|
- echo: Tap General
|
|
115
129
|
- tool: gesture-tap
|
|
116
130
|
args:
|
|
@@ -207,6 +221,7 @@ After applying a correction, re-run `flow-execute` to verify.
|
|
|
207
221
|
Apply these when recording new flows to reduce future breakage:
|
|
208
222
|
|
|
209
223
|
- **Echo expected state, not just actions.** Write `"On Settings > General screen, about to tap About"` not `"Tap About"`. During diagnosis these tell you what the screen _should_ look like.
|
|
224
|
+
- **Gate transitions with `await-ui-element`, not fixed delays.** After a tap that triggers a navigation, record an `await-ui-element` step that waits for the next screen's element to be `visible` (or a spinner to be `hidden`) before the following step. This removes the **Timing** failure mode in §10.2 (the element is in the tree but the tap fired before the screen settled) and is more reliable than `delayMs` or an extra `screenshot`. An unmet wait stops replay at that step, so a mistimed step can never run blind.
|
|
210
225
|
- **Add screenshot steps after critical navigation.** Insert `screenshot` steps after screen transitions. These produce images in the flow result you can inspect during diagnosis.
|
|
211
226
|
- **Write specific executionPrerequisites.** `"App on home tab, user logged in, simulator UDID is <X>"` — not `"App running"`. Verify with `screenshot` + `describe` before acknowledging.
|
|
212
227
|
- **Prefer launch-app / open-url over navigation chains.** Deep links are more resilient to layout changes than tap sequences.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: argent-device-interact
|
|
3
|
-
description: Interact with an iOS simulator, Android emulator, or Chromium (CDP) app using argent MCP tools. Use when tapping UI elements, performing gestures, scrolling/swiping, typing text, pressing hardware buttons, launching apps, opening URLs, taking screenshots, or checking visible app state after interactions.
|
|
3
|
+
description: Interact with an iOS simulator, Android emulator, or Chromium (CDP) app using argent MCP tools. Use when tapping UI elements, performing gestures, scrolling/swiping, typing text, pressing hardware buttons, launching apps, opening URLs, taking screenshots, waiting for an element to appear or disappear, or checking visible app state after interactions.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
## Unified tool surface
|
|
@@ -28,7 +28,7 @@ Use `list-devices` to get a target id. Results are tagged with `platform` (`ios`
|
|
|
28
28
|
1. **Always refer to tapping_rule** from your argent.md rule before tapping.
|
|
29
29
|
2. Before performing interactions, consider whether they can be **dispatched sequentially** - more on that in `run-sequence`.
|
|
30
30
|
3. **Use `gesture-swipe` for lists/scrolling**, not `gesture-custom`, unless you need non-linear movement. On Chromium use `gesture-scroll` instead — `gesture-swipe` is touch-only. Consider whether you need multiple swipes, if yes - use `run-sequence`.
|
|
31
|
-
4. **Tap a text field before typing
|
|
31
|
+
4. **Tap a text field before typing**, then use `keyboard` to enter text.
|
|
32
32
|
5. **Coordinates are normalized** — always 0.0–1.0, not pixels.
|
|
33
33
|
6. **For app navigation, prefer `describe` first.** It works on any screen without app restart. Do not navigate from screenshots on regular in-app screens unless `describe` failed to expose a reliable target. Use `native-describe-screen` only when you need app-scoped UIKit properties.
|
|
34
34
|
|
|
@@ -54,25 +54,25 @@ Common schemes: `messages://`, `settings://`, `maps://?q=<query>`, `tel://<numbe
|
|
|
54
54
|
|
|
55
55
|
## 4. Choosing the Right Tool
|
|
56
56
|
|
|
57
|
-
| Action | Tool
|
|
58
|
-
| ----------------- |
|
|
59
|
-
| Multiple actions | `run-sequence`
|
|
60
|
-
| Open an app | `launch-app`
|
|
61
|
-
| Restart an app | `restart-app`
|
|
62
|
-
| Open URL/scheme | `open-url`
|
|
63
|
-
| Single tap | `gesture-tap`
|
|
64
|
-
| Scroll/swipe | `gesture-swipe`
|
|
65
|
-
| Scroll (Chromium) | `gesture-scroll`
|
|
66
|
-
| Drag (Chromium) | `gesture-drag`
|
|
67
|
-
| Long press | `gesture-custom`
|
|
68
|
-
| Drag & drop | `gesture-custom`
|
|
69
|
-
| Pinch/zoom | `gesture-pinch`
|
|
70
|
-
| Rotation | `gesture-rotate`
|
|
71
|
-
| Custom gesture | `gesture-custom`
|
|
72
|
-
| Hardware key | `button`
|
|
73
|
-
| Type text
|
|
74
|
-
|
|
|
75
|
-
|
|
|
57
|
+
| Action | Tool | Notes |
|
|
58
|
+
| ----------------- | ------------------ | ---------------------------------------------------------------- |
|
|
59
|
+
| Multiple actions | `run-sequence` | Batch steps in one call (no intermediate screenshots) |
|
|
60
|
+
| Open an app | `launch-app` | **Always — never tap home-screen icons** |
|
|
61
|
+
| Restart an app | `restart-app` | Terminate and relaunch by bundle ID |
|
|
62
|
+
| Open URL/scheme | `open-url` | Web pages, deep links, URL schemes |
|
|
63
|
+
| Single tap | `gesture-tap` | Buttons, links, checkboxes |
|
|
64
|
+
| Scroll/swipe | `gesture-swipe` | Straight-line scroll or swipe |
|
|
65
|
+
| Scroll (Chromium) | `gesture-scroll` | Wheel-based; deltas are window fractions, positive deltaY = down |
|
|
66
|
+
| Drag (Chromium) | `gesture-drag` | Sliders, drag-and-drop, text selection |
|
|
67
|
+
| Long press | `gesture-custom` | Context menus, drag start |
|
|
68
|
+
| Drag & drop | `gesture-custom` | Complex drag interactions |
|
|
69
|
+
| Pinch/zoom | `gesture-pinch` | Two-finger pinch with auto-interpolation |
|
|
70
|
+
| Rotation | `gesture-rotate` | Two-finger rotation with auto-interpolation |
|
|
71
|
+
| Custom gesture | `gesture-custom` | Arbitrary touch sequences, optional interpolation |
|
|
72
|
+
| Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton |
|
|
73
|
+
| Type text | `keyboard` | iOS+Android. Supports Enter, Escape, arrows |
|
|
74
|
+
| Rotate device | `rotate` | Orientation changes |
|
|
75
|
+
| Wait for UI | `await-ui-element` | Block until an element is visible/hidden/exists/contains text |
|
|
76
76
|
|
|
77
77
|
## 5. Finding Tap Targets
|
|
78
78
|
|
|
@@ -161,14 +161,6 @@ For long-press, drag-and-drop, and other complex sequences, see `references/gest
|
|
|
161
161
|
|
|
162
162
|
Values: `home`, `back`, `power`, `volumeUp`, `volumeDown`, `appSwitch`, `actionButton`
|
|
163
163
|
|
|
164
|
-
### paste — Type text into focused field (iOS only)
|
|
165
|
-
|
|
166
|
-
```json
|
|
167
|
-
{ "udid": "<UDID>", "text": "Hello, world!" }
|
|
168
|
-
```
|
|
169
|
-
|
|
170
|
-
Tap the field first, then paste. Fall back to `keyboard` if it doesn't work. On Android the call is rejected by the capability gate ("Tool 'paste' is not supported on android") — use `keyboard` directly.
|
|
171
|
-
|
|
172
164
|
### keyboard — Type text or press special keys
|
|
173
165
|
|
|
174
166
|
```json
|
|
@@ -185,6 +177,23 @@ Special keys: `enter`, `escape`, `backspace`, `tab`, `space`, `arrow-up`, `arrow
|
|
|
185
177
|
|
|
186
178
|
Values: `Portrait`, `LandscapeLeft`, `LandscapeRight`, `PortraitUpsideDown`
|
|
187
179
|
|
|
180
|
+
### await-ui-element — Block until a UI element reaches a state
|
|
181
|
+
|
|
182
|
+
Instead of polling `screenshot`/`describe` in a loop, use `await-ui-element` to block server-side until an element reaches an expected state (or `timeoutMs`, default 5000ms, elapses). It polls the same accessibility/DOM tree as `describe`. (For a plain pause, use your own harness sleep — this tool deliberately has no bare-timer mode.)
|
|
183
|
+
|
|
184
|
+
```json
|
|
185
|
+
{ "udid": "<UDID>", "condition": "visible", "selector": { "text": "Continue" } }
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
- `condition`: `exists`, `visible`, `hidden`, or `text`.
|
|
189
|
+
- `selector`: `{ text?, identifier?, role? }` — every provided field must match (case-insensitive substring). `text` matches the element's label or value; `identifier` matches its accessibility id / resource-id / testID; `role` matches its element role (e.g. `AXButton`, `button`, `TextView`, `StaticText`). The synthetic `ROOT` container `describe` prints is never matched, so a `role` like `AXGroup`/`html` won't trivially "match the screen".
|
|
190
|
+
- Prefer a **specific** selector. A loose substring can match several elements, and the tool may then key off one you didn't mean: `text` reads the **first** match in **reading order** (top-to-bottom, left-to-right — the same order `describe` lists them, so it's the one you saw first), while `visible`/`exists` are satisfied by **any** match. Disambiguate with a longer or more exact string, an `identifier`, or a `role` (e.g. pin to a text role like `StaticText` to skip a same-named button). On a `text` timeout the `note` quotes the matched element's text, so you can see which one it landed on.
|
|
191
|
+
- `text` condition also needs `expectedText` (substring the matched element must contain).
|
|
192
|
+
- `hidden` treats a selector that matches **nothing** as already-hidden, so a typo'd selector returns an instant (false) success. Double-check the selector for `hidden` waits — the result `note` flags when the selector never matched any element. (On iOS, if the accessibility backend is down the tree comes back empty; the tool will **not** report `hidden` success off such a degraded read and the `note` surfaces the boot hint instead.)
|
|
193
|
+
- Optional `timeoutMs` (default 5000) and `pollIntervalMs` (default 400).
|
|
194
|
+
|
|
195
|
+
Returns `{ success, elapsed }`; on a timeout `success` is `false` and a `note` explains what was seen.
|
|
196
|
+
|
|
188
197
|
---
|
|
189
198
|
|
|
190
199
|
## 7. Screenshots
|
|
@@ -246,10 +255,12 @@ Use the sequencing when:
|
|
|
246
255
|
|
|
247
256
|
### Allowed tools inside `run-sequence`
|
|
248
257
|
|
|
249
|
-
`gesture-tap`, `gesture-swipe`, `gesture-custom`, `gesture-pinch`, `gesture-rotate`, `button`, `keyboard`, `rotate`
|
|
258
|
+
`gesture-tap`, `gesture-swipe`, `gesture-scroll`, `gesture-drag`, `gesture-custom`, `gesture-pinch`, `gesture-rotate`, `button`, `keyboard`, `rotate`, `await-ui-element`
|
|
250
259
|
|
|
251
260
|
The `udid` is shared — do **not** include it in each step's `args`. Optional `delayMs` per step (default 100ms).
|
|
252
261
|
|
|
262
|
+
Add an `await-ui-element` step to gate a later tap on a screen transition (e.g. tap → wait for the next screen's button → tap it). If its condition is **not** met before the timeout, the sequence stops at that step and the following steps do **not** run — so a mistimed tap can't fire against a screen that never settled.
|
|
263
|
+
|
|
253
264
|
### Examples
|
|
254
265
|
|
|
255
266
|
Scroll down three times:
|
|
@@ -293,7 +304,25 @@ Tap a known button, then scroll down:
|
|
|
293
304
|
}
|
|
294
305
|
```
|
|
295
306
|
|
|
296
|
-
|
|
307
|
+
Tap, wait for the next screen, then act on it — the `await-ui-element` step **gates** the tap after it:
|
|
308
|
+
|
|
309
|
+
```json
|
|
310
|
+
{
|
|
311
|
+
"udid": "<UDID>",
|
|
312
|
+
"steps": [
|
|
313
|
+
{ "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.9 } },
|
|
314
|
+
{
|
|
315
|
+
"tool": "await-ui-element",
|
|
316
|
+
"args": { "condition": "visible", "selector": { "text": "Continue" } }
|
|
317
|
+
},
|
|
318
|
+
{ "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.5 } }
|
|
319
|
+
]
|
|
320
|
+
}
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
Prefer this over a fixed `delayMs` when a step depends on a screen transition: it adapts to real load time, and if the condition is not met before the timeout the sequence **stops there** so the next tap can't fire against a screen that never settled.
|
|
324
|
+
|
|
325
|
+
Stops on the first error (or unmet `await-ui-element` condition) and returns partial results.
|
|
297
326
|
|
|
298
327
|
---
|
|
299
328
|
|
|
@@ -86,14 +86,14 @@ To revisit a previous trace:
|
|
|
86
86
|
|
|
87
87
|
Bottlenecks are categorized by severity:
|
|
88
88
|
|
|
89
|
-
- **RED**: CPU functions taking >15% of total time, all UI hangs,
|
|
90
|
-
- **YELLOW**: CPU functions taking
|
|
89
|
+
- **RED**: CPU functions taking >15% of total time, all UI hangs, and **attributed** memory leaks (those with a resolved responsible frame). These require immediate attention.
|
|
90
|
+
- **YELLOW**: CPU functions taking 3-15% of total time, and **unattributed** memory leaks (`<Call stack limit reached>`, no library — see the memory-leaks caveat below). Worth investigating but may be acceptable.
|
|
91
91
|
|
|
92
92
|
Each bottleneck type indicates a different class of problem:
|
|
93
93
|
|
|
94
94
|
- **CPU hotspots**: Native functions consuming excessive CPU time. Look for tight loops, expensive computations, or redundant work.
|
|
95
95
|
- **UI hangs**: Main thread blocked long enough to cause visible jank or unresponsiveness. Often caused by synchronous I/O, heavy layout passes, or lock contention.
|
|
96
|
-
- **Memory leaks**: Objects allocated but never freed. Common causes include retain cycles, unclosed resources, or forgotten observers.
|
|
96
|
+
- **Memory leaks**: Objects allocated but never freed. Common causes include retain cycles, unclosed resources, or forgotten observers. Argent records via `xctrace --attach`, which has no malloc-stack history, so on the simulator most leaks come back **unattributed** (`<Call stack limit reached>`, no library) and are dominated by benign system allocations — these are reported as a low-confidence YELLOW summary, not confirmed RED leaks. For attributed stacks, capture with malloc stack logging enabled at launch.
|
|
97
97
|
|
|
98
98
|
---
|
|
99
99
|
|
|
@@ -29,7 +29,7 @@ For implementation tasks that modify visible UI, this workflow can also serve as
|
|
|
29
29
|
- **Permission prompts / system modal overlays**: try `describe` first. Fall back to `screenshot` only if the overlay is not exposed reliably.
|
|
30
30
|
- **Fallback**: use `screenshot` to estimate where the desired component is, then verify immediately after the action.
|
|
31
31
|
3. **Interact**: Perform the action (`gesture-tap`, `gesture-swipe`, `keyboard`, `button`, ...) — you receive a screenshot automatically.
|
|
32
|
-
4. **Verify**: Check the returned screenshot for expected results. If it shows a loading/transitional state,
|
|
32
|
+
4. **Verify**: Check the returned screenshot for expected results. If it shows a loading/transitional state, prefer blocking until it settles with `await-ui-element` (expected element `visible`, or a spinner `hidden`) over a guessed delay — but only with a selector you can trust (`text`/`identifier`/`role`) that the screen is known to have or that you saw in a prior `describe`; a guessed one just times out. Otherwise use a short fixed wait. Pick evidence by what's being asserted:
|
|
33
33
|
- **Visual** (layout, spacing, color, typography, image/icon rendering, clipping, overflow, text rendering): prefer `screenshot-diff` against the baseline captured in step 1 — it surfaces pixel-visible changes the auto-screenshot might miss. Fall back to visual inspection of the auto-screenshot only when a stable baseline isn't available.
|
|
34
34
|
- **Structural** (navigation state, element existence, accessibility labels/values, selection, hierarchy, route): verify with `describe`, `debugger-component-tree`, or `native-describe-screen`.
|
|
35
35
|
- **Runtime / log / network** (console errors, API calls, persistence, timing): verify with `view-network-logs`, `debugger-log-registry`, `debugger-evaluate`, or targeted tests.
|
|
@@ -58,9 +58,9 @@ Steps:
|
|
|
58
58
|
```
|
|
59
59
|
1. screenshot → see login screen
|
|
60
60
|
2. gesture-tap { x: 0.5, y: 0.4 } → tap email field
|
|
61
|
-
3.
|
|
61
|
+
3. keyboard { text: "user@example.com" }
|
|
62
62
|
4. gesture-tap { x: 0.5, y: 0.55 } → tap password field
|
|
63
|
-
5.
|
|
63
|
+
5. keyboard { text: "password123" }
|
|
64
64
|
6. gesture-tap { x: 0.5, y: 0.7 } → tap Login button
|
|
65
65
|
7. screenshot → verify home screen appeared
|
|
66
66
|
```
|
|
@@ -89,11 +89,20 @@ Steps:
|
|
|
89
89
|
8. Report combined verdict from expected behavior, visual inspection, diff summary, and structural evidence.
|
|
90
90
|
```
|
|
91
91
|
|
|
92
|
+
### Wait for a loading spinner
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
1. gesture-tap { x: 0.5, y: 0.7 } → trigger an action that fetches data
|
|
96
|
+
2. screenshot → loading spinner is showing
|
|
97
|
+
3. await-ui-element { condition: hidden, selector: { text: "Loading" } } → block until the fetch finishes and the spinner disappears
|
|
98
|
+
4. describe / screenshot → verify the fetched content rendered
|
|
99
|
+
```
|
|
100
|
+
|
|
92
101
|
---
|
|
93
102
|
|
|
94
103
|
## 4. Recovery Pattern
|
|
95
104
|
|
|
96
|
-
- If
|
|
105
|
+
- If a screen is mid-transition or loading: block until it settles with `await-ui-element` (wait for the target element to be `visible`, or the spinner/placeholder to be `hidden`) instead of a blind fixed delay, then re-check. Fall back to a fixed wait + `screenshot` only when no element reliably marks the transition.
|
|
97
106
|
- If tap misses target: re-run discovery tool (`describe` / `debugger-component-tree`), retry once with new coordinates.
|
|
98
107
|
- If a permission dialog or modal is visible: re-run `describe` first. Stay in screenshot-driven navigation only when the overlay is not exposed reliably, then switch back to `describe` / `debugger-component-tree` as soon as it is dismissed.
|
|
99
108
|
- If tap fails twice at same coordinates: stop, re-discover, report if element not found.
|
|
@@ -101,7 +110,7 @@ Steps:
|
|
|
101
110
|
|
|
102
111
|
## Tips
|
|
103
112
|
|
|
104
|
-
- **
|
|
113
|
+
- **Wait on the UI, don't poll.** When a step needs the screen to change first, gate it with `await-ui-element` (block until an element is `visible`/`hidden` or contains `text`) rather than repeated `screenshot` calls with fixed sleeps. See the `await-ui-element` section of `argent-device-interact`.
|
|
105
114
|
- **Use `gesture-custom` for long-press** context menus (800ms hold).
|
|
106
115
|
- **Report clearly**: state what you expected, what you saw, and the verdict.
|
|
107
116
|
- **Permission modals**: try `describe` first. Use `screenshot` only as fallback, tap one visible button at a time, and verify with the returned screenshot before continuing.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: argent-vega
|
|
3
|
+
description: Control and inspect Amazon Fire TV (Vega) apps via argent — launch/restart/reinstall apps, read the on-screen element tree, navigate with the D-pad remote, type, and screenshot. Use when the task mentions Vega, Fire TV, or VVD, or involves driving a Vega virtual device.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Argent Vega (Amazon Fire TV)
|
|
7
|
+
|
|
8
|
+
## Critical
|
|
9
|
+
|
|
10
|
+
- Vega is a TV platform
|
|
11
|
+
- **D-pad only.** Drive every interaction with `tv-remote`. Never use `gesture-*` / touch — they are unsupported on Vega.
|
|
12
|
+
- **Always `describe` before navigating.** Find the live cursor from the tree — the `[focused]` element, or `[selected]` when nothing reports `[focused]` (the toolkit often marks the highlighted item `[selected]` while `focused` stays false). Never guess focus position from a screenshot.
|
|
13
|
+
- All tools take the Vega `serial` (from `list-devices`) as `udid`.
|
|
14
|
+
|
|
15
|
+
## The navigation loop
|
|
16
|
+
|
|
17
|
+
Per screen, two calls:
|
|
18
|
+
|
|
19
|
+
1. `describe` — find the cursor (`[focused]`, or `[selected]` if no `[focused]`) and your target.
|
|
20
|
+
2. Compute the full D-pad path from focus → target (count rows/columns from the frames) and fire it as **one** `tv-remote {button:[...]}` ending in `select`.
|
|
21
|
+
|
|
22
|
+
Then `describe` again to confirm. On a miss, run the loop again.
|
|
23
|
+
|
|
24
|
+
## Tools
|
|
25
|
+
|
|
26
|
+
### Device lifecycle
|
|
27
|
+
|
|
28
|
+
- `list-devices` → Vega devices appear with a `serial` (use as `udid`) and a `vvdImage`. Start here to get both.
|
|
29
|
+
- `boot-device {vvdImage}` — starts the single SDK-managed VVD (e.g. `vvdImage:"tv"`) and returns its `serial`. Skip if `list-devices` already shows a running device.
|
|
30
|
+
- **Stopping the VVD** — run `vega virtual-device stop` in your shell.
|
|
31
|
+
|
|
32
|
+
### App lifecycle
|
|
33
|
+
|
|
34
|
+
- `launch-app {udid, bundleId}` — `bundleId` = interactive component app id from manifest.toml (e.g. `com.example.app.main`)
|
|
35
|
+
- `restart-app {udid, bundleId}` — terminate + launch
|
|
36
|
+
- `reinstall-app {udid, bundleId, appPath}` — uninstall + install; `appPath` = a `.vpkg`
|
|
37
|
+
- `describe {udid}` → on-screen element tree. The discovery tool — call before navigating
|
|
38
|
+
- `tv-remote {udid, button}` — D-pad; single key, path array, or `repeat`
|
|
39
|
+
- `keyboard {udid, text}` or `{udid, key:"enter"}` — focus the field with the D-pad first
|
|
40
|
+
- `screenshot {udid, scale?}` — captured host-side via `adb`
|
|
41
|
+
|
|
42
|
+
### `describe`
|
|
43
|
+
|
|
44
|
+
Nested element tree from the on-device automation toolkit — each line is a `button`/`text`/`image` with its label, `id` (test_id), `[clickable]`, and **`[focused]`/`[selected]`** + a normalized [0,1] frame. `[focused]` is the live D-pad cursor when present; in practice the toolkit usually leaves `focused` false and marks the highlighted item `[selected]`, so treat `[selected]` as the cursor whenever no element reports `[focused]`. Navigate on the tree alone. If the tree comes back empty → `restart-app` and retry.
|
|
45
|
+
|
|
46
|
+
### `tv-remote`
|
|
47
|
+
|
|
48
|
+
`button` is a single key **or a whole path**. Keys: `up`/`down`/`left`/`right`, `select`, `back`, `home`, `menu`, `playPause`, `rewind`, `fastForward`. Single: `{button:"down"}`. Repeat one key: `{button:"down", repeat:3}`. Whole path in one call: `{button:["up","right","right","select"]}` — strongly prefer this for any multi-step move.
|
|
49
|
+
|
|
50
|
+
## Fast Refresh
|
|
51
|
+
|
|
52
|
+
Needs a Debug build + Metro running. argent only connects to Metro — it does not start Metro or port-forward (any platform); do these in your shell.
|
|
53
|
+
|
|
54
|
+
1. Build a **Debug** `.vpkg` and install it: `vega device install-app -p <path/to/debug.vpkg>`
|
|
55
|
+
2. `npm start` (Metro on :8081; use `npm start`, not `npx react-native start`)
|
|
56
|
+
3. `vega device start-port-forwarding --port 8081 --forward false` (reverse)
|
|
57
|
+
4. `vega device launch-app -a <appId>`
|
|
58
|
+
|
|
59
|
+
Metro must be up before launch; confirm `http://localhost:8081/json/list` lists a `Hermes React Native` target. Then `.tsx` edits hot-reload live.
|
|
60
|
+
|
|
61
|
+
## Troubleshooting
|
|
62
|
+
|
|
63
|
+
- **`describe` returns an empty tree** → `restart-app` (the automation toolkit attaches at launch), then retry.
|
|
64
|
+
- **Keyboard / D-pad input is ignored** → enable developer mode inside the VVD: `vsm developer-mode enable`.
|
|
65
|
+
- **Editing `node_modules` has no effect** → you are on a Release build. Release Vega apps load JavaScript and native code split and stored on device, so patching `node_modules` only works in Debug builds.
|
|
66
|
+
|
|
67
|
+
## Platform notes
|
|
68
|
+
|
|
69
|
+
- Metro connects only on port **8081** — fixed, cannot be changed.
|
|
70
|
+
- Profiling / crashes → use the `amazon-devices-buildertools-mcp` server (`analyze_perfetto_traces`, `get_app_hot_functions`, `symbolicate_acr`).
|
|
71
|
+
- Unsupported tools, with the Vega equivalent: `gesture-*` → use `tv-remote`; `open-url` → not wired; `debugger-*` → JS debugger not supported on Vega. These fail with `Tool '<id>' is not supported on vega vvd.` (or `... is not yet implemented on vega.`).
|
|
72
|
+
|
|
73
|
+
## Knowledgebase
|
|
74
|
+
|
|
75
|
+
- Search Vega docs with the `search_documentation` tool (`amazon-devices-buildertools-mcp` server).
|
|
76
|
+
- Community Q&A at community.amazondeveloper.com.
|