agent-device 0.20.5 → 0.20.7

package/README.md

@@ -15,7 +15,7 @@
 
 Let your coding agent verify its changes in the running app.
 
-`agent-device` lets coding agents inspect, control, and verify apps on iOS, Android, tvOS, Android TV, Amazon Vega OS TV through the Vega Virtual Device (VVD), web, macOS, and Linux. Agents can read token-efficient accessibility snapshots where supported, find elements by ref or selector, run device actions, and save evidence for review. Initial Vega OS support is VVD-only and covers discovery, app lifecycle, and complete TV-remote control; physical Fire TV, capture, and selector backends remain unsupported.
+`agent-device` lets coding agents inspect, control, and verify apps on iOS, Android, HarmonyOS, tvOS, Android TV, Amazon Vega OS TV through the Vega Virtual Device (VVD), web, macOS, and Linux. Agents can read token-efficient accessibility snapshots where supported, find elements by ref or selector, run device actions, and save evidence for review. HarmonyOS support uses HDC and ArkUI `uitest` on connected devices and DevEco emulators; use `capabilities --platform harmonyos` to inspect its evidence-backed command subset. Initial Vega OS support is VVD-only and covers discovery, app lifecycle, and complete TV-remote control; physical Fire TV, capture, and selector backends remain unsupported.
 
 Your coding agent or QA tool reads each result and chooses the next command. `agent-device` runs the command and saves evidence when asked.
 
@@ -61,6 +61,8 @@ agent-device close
 
 Use refs only from the latest output. Do not assume an earlier `@eN` still identifies the same element. After a command with `--settle`, use the refs in its diff. Take another snapshot only if the diff omits what you need.
 
+`--settle` works the same way on `scroll` and `back`, so scroll-then-observe and back-then-observe are one call too.
+
 Snapshots use the app's accessibility tree. Clear labels, roles, and test IDs make agent runs more reliable. Use screenshots and videos as evidence or when accessibility data is poor. Use refs and selectors for actions and assertions when you can.
 
 ![agent-device demo showing Codex using agent-device to create a new contact in the iOS Contacts app from a simple prompt](./website/docs/public/agent-device-contacts.gif)
@@ -107,7 +109,7 @@ See [Commands](https://oss.callstack.com/agent-device/docs/commands) for the com
 
 ## How it works
 
-`agent-device` keeps device state in sessions. It sends commands to XCTest on iOS and tvOS, ADB and the snapshot helper on Android, Vega CLI/VDA on the Vega Virtual Device, a local helper on macOS, and AT-SPI on Linux.
+`agent-device` keeps device state in sessions. It sends commands to XCTest on iOS and tvOS, ADB and the snapshot helper on Android, HDC and ArkUI `uitest` on HarmonyOS, Vega CLI/VDA on the Vega Virtual Device, a local helper on macOS, and AT-SPI on Linux.
 
 Node.js apps can use the typed client or public subpaths. `agent-device/android-adb` provides the Android ADB provider interface, helpers for logcat, the clipboard, the keyboard, and apps, and port reverse management.
 
@@ -115,7 +117,7 @@ Node.js apps can use the typed client or public subpaths. `agent-device/android-
 
 ### What is agent-device?
 
-`agent-device` is a command-line tool that lets coding agents inspect, control, and verify apps and save evidence for review. It supports iOS, Android, TV, web, macOS, and Linux.
+`agent-device` is a command-line tool that lets coding agents inspect, control, and verify apps and save evidence for review. It supports iOS, Android, HarmonyOS, TV, web, macOS, and Linux.
 
 ### Does it work with React Native, Expo, Flutter, and native apps?
 

package/android/ime-helper/dist/agent-device-android-ime-helper-0.20.7.apk.sha256

@@ -0,0 +1 @@
+7311d6a527bdf8be33b752e59e73f4a527585654f8d530d3680ab15ccf271cb0  agent-device-android-ime-helper-0.20.7.apk

package/android/ime-helper/dist/{agent-device-android-ime-helper-0.20.5.manifest.json → agent-device-android-ime-helper-0.20.7.manifest.json}

@@ -1,10 +1,10 @@
 {
   "name": "android-ime-helper",
-  "version": "0.20.5",
-  "assetName": "agent-device-android-ime-helper-0.20.5.apk",
-  "sha256": "1f5a95fcf0c6d668a11453644b2e897147932b528c1aa6c74883eaf9d73a5e7f",
+  "version": "0.20.7",
+  "assetName": "agent-device-android-ime-helper-0.20.7.apk",
+  "sha256": "7311d6a527bdf8be33b752e59e73f4a527585654f8d530d3680ab15ccf271cb0",
   "packageName": "com.callstack.agentdevice.imehelper",
-  "versionCode": 20005,
+  "versionCode": 20007,
   "serviceComponent": "com.callstack.agentdevice.imehelper/.TestInputMethodService",
   "broadcastProtocol": "android-ime-helper-v1"
 }

package/android/snapshot-helper/dist/agent-device-android-snapshot-helper-0.20.7.apk.sha256

@@ -0,0 +1 @@
+c45f72eea93c19b4a17fedb4b25f880d5e225c9ab41be4017db8613fdd276e09  agent-device-android-snapshot-helper-0.20.7.apk

package/android/snapshot-helper/dist/{agent-device-android-snapshot-helper-0.20.5.manifest.json → agent-device-android-snapshot-helper-0.20.7.manifest.json}

@@ -1,13 +1,13 @@
 {
   "name": "android-snapshot-helper",
-  "version": "0.20.5",
-  "releaseTag": "v0.20.5",
-  "assetName": "agent-device-android-snapshot-helper-0.20.5.apk",
+  "version": "0.20.7",
+  "releaseTag": "v0.20.7",
+  "assetName": "agent-device-android-snapshot-helper-0.20.7.apk",
   "apkUrl": null,
-  "sha256": "ce370af9f1c97e7c812cb93772f864596731c10f3d0b7cadc1c88b772a062970",
-  "checksumName": "agent-device-android-snapshot-helper-0.20.5.apk.sha256",
+  "sha256": "c45f72eea93c19b4a17fedb4b25f880d5e225c9ab41be4017db8613fdd276e09",
+  "checksumName": "agent-device-android-snapshot-helper-0.20.7.apk.sha256",
   "packageName": "com.callstack.agentdevice.snapshothelper",
-  "versionCode": 20005,
+  "versionCode": 20007,
   "instrumentationRunner": "com.callstack.agentdevice.snapshothelper/.SnapshotInstrumentation",
   "minSdk": 23,
   "targetSdk": 36,

package/dist/apple/runner/AgentDeviceRunner/AgentDeviceRunner/AgentDeviceRunnerApp.m

@@ -62,6 +62,13 @@ int main(int argc, const char *argv[]) {
 
 @implementation AgentDeviceRunnerViewController
 
+- (void)agentDeviceTextEntryDidChange:(UITextField *)textField {
+  if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-disappear-after-input"] &&
+      textField.text.length > 0) {
+    [textField removeFromSuperview];
+  }
+}
+
 - (void)viewDidLoad {
   [super viewDidLoad];
 
@@ -78,6 +85,47 @@ int main(int argc, const char *argv[]) {
     [label.centerXAnchor constraintEqualToAnchor:self.view.centerXAnchor],
     [label.centerYAnchor constraintEqualToAnchor:self.view.centerYAnchor],
   ]];
+
+  // Keep the fixture behind a launch argument so normal runner snapshots remain unchanged.
+#if TARGET_OS_IOS
+  if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-regression"]) {
+    UITextField *textField = [[UITextField alloc] init];
+    textField.accessibilityIdentifier = @"agent-device-hardware-keyboard-input";
+    textField.borderStyle = UITextBorderStyleRoundedRect;
+    textField.inputView = [[UIView alloc] initWithFrame:CGRectMake(0, 0, 1, 1)];
+    [textField addTarget:self
+                  action:@selector(agentDeviceTextEntryDidChange:)
+        forControlEvents:UIControlEventEditingChanged];
+    textField.translatesAutoresizingMaskIntoConstraints = NO;
+    [self.view addSubview:textField];
+    [NSLayoutConstraint activateConstraints:@[
+      [textField.centerXAnchor constraintEqualToAnchor:self.view.centerXAnchor],
+      [textField.topAnchor constraintEqualToAnchor:label.bottomAnchor constant:24],
+      [textField.widthAnchor constraintEqualToConstant:240],
+      [textField.heightAnchor constraintEqualToConstant:44],
+    ]];
+  }
+
+  if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-selector-read-regression"]) {
+    NSString *const duplicateIdentifier = @"agent-device-selector-read-duplicate";
+
+    UIButton *visibleButton = [UIButton buttonWithType:UIButtonTypeSystem];
+    visibleButton.accessibilityIdentifier = duplicateIdentifier;
+    [visibleButton setTitle:@"Readable target" forState:UIControlStateNormal];
+    visibleButton.translatesAutoresizingMaskIntoConstraints = NO;
+    [self.view addSubview:visibleButton];
+
+    UILabel *offscreenLabel = [[UILabel alloc] initWithFrame:CGRectMake(-200, -200, 100, 40)];
+    offscreenLabel.accessibilityIdentifier = duplicateIdentifier;
+    offscreenLabel.text = @"Decorative duplicate";
+    [self.view addSubview:offscreenLabel];
+
+    [NSLayoutConstraint activateConstraints:@[
+      [visibleButton.centerXAnchor constraintEqualToAnchor:self.view.centerXAnchor],
+      [visibleButton.topAnchor constraintEqualToAnchor:label.bottomAnchor constant:24],
+    ]];
+  }
+#endif
 }
 
 @end

package/dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/AgentDeviceRunnerUITests-Bridging-Header.h

@@ -1,3 +1,4 @@
 #import "RunnerObjCExceptionCatcher.h"
 #import "RunnerAXSnapshotBridge.h"
 #import "RunnerSynthesizedGesture.h"
+#import "RunnerSynthesizedTextEntry.h"

package/dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerAXSnapshotBridge.h

@@ -1,16 +1,123 @@
 #import <Foundation/Foundation.h>
+#import <CoreGraphics/CoreGraphics.h>
 #import <XCTest/XCTest.h>
 
 NS_ASSUME_NONNULL_BEGIN
 
+/// Keys of the `deepExtension` dictionary in the snapshot response, shared
+/// with the Swift consumer so the response shape has one spelling.
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionCallsKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionNodesAddedKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionPendingKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotDeepExtensionMissedKey;
+
+/// Keys of the `customActions` dictionary in the snapshot response: how many
+/// merged elements were eligible for an action read, and how many the bounded
+/// pass actually reached. Present only when the capture asked for actions.
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotCustomActionsKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotCustomActionsReadKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotCustomActionsCandidatesKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotCustomActionsTruncatedKey;
+FOUNDATION_EXPORT NSString *const RunnerAXSnapshotCustomActionsBlockedKey;
+
+/// A depth-capped childless node awaiting an element-rooted follow-up request.
+/// Public so the runner unit bundle can drive the extension's miss paths with
+/// fabricated snapshots — the executed contract that missed frontiers are
+/// counted, never silently dropped.
+@interface RunnerAXSnapshotFrontier : NSObject
+@property(nonatomic, strong, nullable) id snapshot;
+@property(nonatomic, strong, nullable) NSMutableDictionary *node;
+@property(nonatomic, assign) NSInteger depth;
+@end
+
 @interface RunnerAXSnapshotBridge : NSObject
 
+/// Captures the app's accessibility tree, then chains element-rooted follow-up
+/// requests from depth-capped frontier nodes (the AX depth limit is
+/// per-request, so re-rooting reaches content the app-rooted request could
+/// not). The response gains a `deepExtension` {calls, nodesAdded,
+/// pendingFrontiers, missedFrontiers} dictionary when any frontier existed and
+/// the call limit allowed extension. Pass 0 to disable extension entirely.
+///
+/// `customActionLimit` bounds the per-element custom-action reads described on
+/// `customActionNamesForElement:axClient:`; pass 0 to skip them entirely.
 + (NSDictionary<NSString *, id> *)snapshotTreeForApplication:(XCUIApplication *)application
                                                     maxDepth:(NSInteger)maxDepth
-                                                    maxNodes:(NSInteger)maxNodes;
+                                                    maxNodes:(NSInteger)maxNodes
+                                      deepExtensionCallLimit:(NSInteger)deepExtensionCallLimit
+                                           customActionLimit:(NSInteger)customActionLimit
+                                                    deadline:(nullable NSDate *)deadline;
+
+/// Names of the element's UIAccessibilityCustomActions, or nil when it has
+/// none.
+///
+/// One element per call on purpose. The bulk snapshot request cannot carry this
+/// attribute: adding it makes testmanagerd's reply decoder reject the nested
+/// arrays a custom action serializes into ("value for key 'NS.objects' was of
+/// unexpected class 'NSArray'"), drop the reply, and time the request out after
+/// ~65s instead of the usual ~110ms. Per-element reads use a different XPC
+/// message and answer normally.
+///
+/// Reading is all we can do: the actions are not invocable from the runner.
+/// XCUIElement exposes no custom-action API; XCAXClient_iOS's
+/// performAction:onElement:value: answers kAXErrorCannotComplete for every AX
+/// action number; setAttribute: on the custom-actions attribute reports success
+/// and does nothing; parameterizedAttribute: answers kAXErrorNoValue; and the
+/// raw AXUIElement C API cannot be aimed at app elements because
+/// XCAccessibilityElement is token-based and answers nil for AXUIElement.
++ (nullable NSArray<NSString *> *)customActionNamesForElement:(id)element
+                                                    axClient:(id)axClient
+                                                   completed:(BOOL *)completed;
+
+/// Bounds one element's contribution to the response (max actions, max name
+/// length). Reports whether anything was clipped so truncation is disclosed
+/// rather than silently presented as a complete list.
++ (NSArray<NSString *> *)cappedActionNames:(NSArray<NSString *> *)names
+                                 truncated:(BOOL *)truncated;
+
+/// The bounded read pass over merged leaves, exposed for the runner unit
+/// bundle: the containment contract (a hung read blocks the pass and is
+/// disclosed, and adds no further dispatches) is only executable from here.
+/// Returns {read, candidates, truncated, blocked}.
++ (NSDictionary<NSString *, NSNumber *> *)
+    annotateCustomActionsOnMergedLeaves:(nullable NSArray<RunnerAXSnapshotFrontier *> *)leaves
+                               axClient:(id)axClient
+                                  limit:(NSInteger)limit
+                              rootFrame:(CGRect)rootFrame
+                               deadline:(nullable NSDate *)deadline;
+
+/// Reads dispatched but not yet returned. The AX call cannot be cancelled, so a
+/// read that blew its deadline stays counted here until it finally returns —
+/// which is precisely the state in which further reads must be refused rather
+/// than queued behind it. Consumed by the read pass to disclose the skip.
++ (NSInteger)customActionReadsInFlight;
+
+/// Total reads ever dispatched. The containment invariant is "a blocked capture
+/// adds no work", which is only observable as this counter standing still.
++ (NSInteger)customActionReadDispatchCount;
+
+/// The shared AX client (`XCUIDevice.accessibilityInterface`), or nil when the
+/// private interface is unavailable.
++ (nullable id)accessibilityClient;
 
 + (NSInteger)processIdentifierForApplication:(XCUIApplication *)application;
 
+/// The frontier-extension loop, exposed for the runner unit bundle: a frontier
+/// whose element vanished or whose re-rooted request fails must count as
+/// missed (see the deepExtension keys above) — an all-miss extension reporting
+/// itself drained would present a capped capture as complete.
++ (nullable NSDictionary *)extendSnapshotFrontiers:(NSMutableArray<RunnerAXSnapshotFrontier *> *)frontiers
+                                          axClient:(id)axClient
+                                        attributes:(NSArray *)attributes
+                                          maxDepth:(NSInteger)maxDepth
+                                          maxNodes:(NSInteger)maxNodes
+                                         nodeCount:(NSInteger *)nodeCount
+                                         truncated:(BOOL *)truncated
+                                      callsAllowed:(NSInteger)callsAllowed
+                                      mergedLeaves:(nullable NSMutableArray<RunnerAXSnapshotFrontier *> *)mergedLeaves
+                                          deadline:(nullable NSDate *)deadline;
+
 @end
 
 NS_ASSUME_NONNULL_END