agent-device 0.10.2 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (203) hide show
  1. package/README.md +6 -0
  2. package/dist/src/36.js +3 -0
  3. package/dist/src/bin.js +82 -66
  4. package/dist/src/daemon.js +40 -39
  5. package/dist/src/index.d.ts +567 -5
  6. package/dist/src/index.js +3 -1
  7. package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift +84 -16
  8. package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift +140 -50
  9. package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Lifecycle.swift +13 -2
  10. package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift +22 -9
  11. package/ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift +221 -0
  12. package/ios-runner/README.md +17 -2
  13. package/ios-runner/RUNNER_PROTOCOL.md +73 -0
  14. package/package.json +11 -6
  15. package/skills/agent-device/SKILL.md +38 -12
  16. package/skills/agent-device/references/bootstrap-install.md +55 -15
  17. package/skills/agent-device/references/exploration.md +65 -8
  18. package/skills/agent-device/references/macos-desktop.md +1 -2
  19. package/skills/agent-device/references/remote-tenancy.md +15 -2
  20. package/skills/agent-device/references/verification.md +7 -3
  21. package/dist/src/224.js +0 -2
  22. package/dist/src/331.js +0 -3
  23. package/dist/src/425.js +0 -1
  24. package/dist/src/bin.d.ts +0 -1
  25. package/dist/src/cli-client-commands.d.ts +0 -8
  26. package/dist/src/cli.d.ts +0 -6
  27. package/dist/src/client-metro.d.ts +0 -64
  28. package/dist/src/client-normalizers.d.ts +0 -20
  29. package/dist/src/client-shared.d.ts +0 -20
  30. package/dist/src/client-types.d.ts +0 -269
  31. package/dist/src/client.d.ts +0 -5
  32. package/dist/src/core/app-events.d.ts +0 -8
  33. package/dist/src/core/batch.d.ts +0 -17
  34. package/dist/src/core/capabilities.d.ts +0 -3
  35. package/dist/src/core/click-button.d.ts +0 -20
  36. package/dist/src/core/dispatch-payload.d.ts +0 -1
  37. package/dist/src/core/dispatch-resolve.d.ts +0 -29
  38. package/dist/src/core/dispatch-series.d.ts +0 -7
  39. package/dist/src/core/dispatch.d.ts +0 -37
  40. package/dist/src/core/open-target.d.ts +0 -4
  41. package/dist/src/core/session-surface.d.ts +0 -3
  42. package/dist/src/core/settings-contract.d.ts +0 -9
  43. package/dist/src/daemon/action-utils.d.ts +0 -3
  44. package/dist/src/daemon/android-system-dialog.d.ts +0 -11
  45. package/dist/src/daemon/app-log-android.d.ts +0 -4
  46. package/dist/src/daemon/app-log-ios.d.ts +0 -7
  47. package/dist/src/daemon/app-log-process.d.ts +0 -15
  48. package/dist/src/daemon/app-log-stream.d.ts +0 -19
  49. package/dist/src/daemon/app-log.d.ts +0 -28
  50. package/dist/src/daemon/artifact-archive.d.ts +0 -12
  51. package/dist/src/daemon/artifact-download.d.ts +0 -12
  52. package/dist/src/daemon/artifact-materialization.d.ts +0 -17
  53. package/dist/src/daemon/artifact-registry.d.ts +0 -12
  54. package/dist/src/daemon/config.d.ts +0 -16
  55. package/dist/src/daemon/context.d.ts +0 -25
  56. package/dist/src/daemon/device-ready.d.ts +0 -6
  57. package/dist/src/daemon/handlers/find.d.ts +0 -40
  58. package/dist/src/daemon/handlers/install-source.d.ts +0 -44
  59. package/dist/src/daemon/handlers/interaction-common.d.ts +0 -41
  60. package/dist/src/daemon/handlers/interaction-flags.d.ts +0 -4
  61. package/dist/src/daemon/handlers/interaction-get.d.ts +0 -3
  62. package/dist/src/daemon/handlers/interaction-is.d.ts +0 -3
  63. package/dist/src/daemon/handlers/interaction-read.d.ts +0 -14
  64. package/dist/src/daemon/handlers/interaction-scroll.d.ts +0 -3
  65. package/dist/src/daemon/handlers/interaction-selector.d.ts +0 -27
  66. package/dist/src/daemon/handlers/interaction-snapshot.d.ts +0 -8
  67. package/dist/src/daemon/handlers/interaction-targeting.d.ts +0 -28
  68. package/dist/src/daemon/handlers/interaction-touch.d.ts +0 -45
  69. package/dist/src/daemon/handlers/interaction.d.ts +0 -9
  70. package/dist/src/daemon/handlers/lease.d.ts +0 -8
  71. package/dist/src/daemon/handlers/parse-utils.d.ts +0 -3
  72. package/dist/src/daemon/handlers/record-trace-android.d.ts +0 -18
  73. package/dist/src/daemon/handlers/record-trace-ios.d.ts +0 -52
  74. package/dist/src/daemon/handlers/record-trace-recording.d.ts +0 -32
  75. package/dist/src/daemon/handlers/record-trace.d.ts +0 -10
  76. package/dist/src/daemon/handlers/session-batch.d.ts +0 -2
  77. package/dist/src/daemon/handlers/session-close.d.ts +0 -31
  78. package/dist/src/daemon/handlers/session-deploy.d.ts +0 -37
  79. package/dist/src/daemon/handlers/session-device-utils.d.ts +0 -26
  80. package/dist/src/daemon/handlers/session-open-target.d.ts +0 -3
  81. package/dist/src/daemon/handlers/session-open.d.ts +0 -22
  82. package/dist/src/daemon/handlers/session-perf.d.ts +0 -2
  83. package/dist/src/daemon/handlers/session-replay-heal.d.ts +0 -8
  84. package/dist/src/daemon/handlers/session-replay-script.d.ts +0 -3
  85. package/dist/src/daemon/handlers/session-runtime-command.d.ts +0 -9
  86. package/dist/src/daemon/handlers/session-runtime.d.ts +0 -36
  87. package/dist/src/daemon/handlers/session-startup-metrics.d.ts +0 -11
  88. package/dist/src/daemon/handlers/session.d.ts +0 -50
  89. package/dist/src/daemon/handlers/snapshot-alert.d.ts +0 -13
  90. package/dist/src/daemon/handlers/snapshot-capture.d.ts +0 -34
  91. package/dist/src/daemon/handlers/snapshot-session.d.ts +0 -15
  92. package/dist/src/daemon/handlers/snapshot-settings.d.ts +0 -24
  93. package/dist/src/daemon/handlers/snapshot-wait.d.ts +0 -37
  94. package/dist/src/daemon/handlers/snapshot.d.ts +0 -16
  95. package/dist/src/daemon/http-server.d.ts +0 -26
  96. package/dist/src/daemon/install-source-resolution.d.ts +0 -5
  97. package/dist/src/daemon/is-predicates.d.ts +0 -15
  98. package/dist/src/daemon/lease-context.d.ts +0 -9
  99. package/dist/src/daemon/lease-registry.d.ts +0 -63
  100. package/dist/src/daemon/materialized-path-registry.d.ts +0 -15
  101. package/dist/src/daemon/network-log.d.ts +0 -32
  102. package/dist/src/daemon/record-trace-errors.d.ts +0 -6
  103. package/dist/src/daemon/recording-gestures.d.ts +0 -3
  104. package/dist/src/daemon/recording-telemetry.d.ts +0 -20
  105. package/dist/src/daemon/recording-timing.d.ts +0 -24
  106. package/dist/src/daemon/request-cancel.d.ts +0 -9
  107. package/dist/src/daemon/request-lock-policy.d.ts +0 -2
  108. package/dist/src/daemon/request-router.d.ts +0 -23
  109. package/dist/src/daemon/runtime-hints.d.ts +0 -19
  110. package/dist/src/daemon/script-utils.d.ts +0 -28
  111. package/dist/src/daemon/scroll-planner.d.ts +0 -12
  112. package/dist/src/daemon/selectors-build.d.ts +0 -5
  113. package/dist/src/daemon/selectors-match.d.ts +0 -6
  114. package/dist/src/daemon/selectors-parse.d.ts +0 -29
  115. package/dist/src/daemon/selectors-resolve.d.ts +0 -33
  116. package/dist/src/daemon/selectors.d.ts +0 -5
  117. package/dist/src/daemon/server-lifecycle.d.ts +0 -23
  118. package/dist/src/daemon/session-open-script.d.ts +0 -7
  119. package/dist/src/daemon/session-routing.d.ts +0 -3
  120. package/dist/src/daemon/session-selector.d.ts +0 -10
  121. package/dist/src/daemon/session-store.d.ts +0 -33
  122. package/dist/src/daemon/snapshot-diff.d.ts +0 -20
  123. package/dist/src/daemon/snapshot-processing.d.ts +0 -10
  124. package/dist/src/daemon/touch-reference-frame.d.ts +0 -7
  125. package/dist/src/daemon/transport.d.ts +0 -6
  126. package/dist/src/daemon/types.d.ts +0 -173
  127. package/dist/src/daemon/upload-registry.d.ts +0 -7
  128. package/dist/src/daemon/upload.d.ts +0 -5
  129. package/dist/src/daemon-client.d.ts +0 -40
  130. package/dist/src/daemon.d.ts +0 -1
  131. package/dist/src/platforms/android/adb.d.ts +0 -5
  132. package/dist/src/platforms/android/app-lifecycle.d.ts +0 -31
  133. package/dist/src/platforms/android/device-input-state.d.ts +0 -19
  134. package/dist/src/platforms/android/devices.d.ts +0 -26
  135. package/dist/src/platforms/android/index.d.ts +0 -8
  136. package/dist/src/platforms/android/input-actions.d.ts +0 -17
  137. package/dist/src/platforms/android/install-artifact.d.ts +0 -11
  138. package/dist/src/platforms/android/manifest.d.ts +0 -1
  139. package/dist/src/platforms/android/notifications.d.ts +0 -11
  140. package/dist/src/platforms/android/open-target.d.ts +0 -4
  141. package/dist/src/platforms/android/screenshot.d.ts +0 -16
  142. package/dist/src/platforms/android/sdk.d.ts +0 -2
  143. package/dist/src/platforms/android/settings.d.ts +0 -3
  144. package/dist/src/platforms/android/snapshot.d.ts +0 -7
  145. package/dist/src/platforms/android/ui-hierarchy.d.ts +0 -21
  146. package/dist/src/platforms/appearance.d.ts +0 -2
  147. package/dist/src/platforms/boot-diagnostics.d.ts +0 -14
  148. package/dist/src/platforms/install-source.d.ts +0 -29
  149. package/dist/src/platforms/ios/app-filter.d.ts +0 -2
  150. package/dist/src/platforms/ios/apps.d.ts +0 -34
  151. package/dist/src/platforms/ios/config.d.ts +0 -10
  152. package/dist/src/platforms/ios/devicectl.d.ts +0 -13
  153. package/dist/src/platforms/ios/devices.d.ts +0 -40
  154. package/dist/src/platforms/ios/ensure-simulator.d.ts +0 -18
  155. package/dist/src/platforms/ios/index.d.ts +0 -3
  156. package/dist/src/platforms/ios/install-artifact.d.ts +0 -18
  157. package/dist/src/platforms/ios/launch-diagnostics.d.ts +0 -11
  158. package/dist/src/platforms/ios/macos-apps.d.ts +0 -12
  159. package/dist/src/platforms/ios/macos-helper.d.ts +0 -69
  160. package/dist/src/platforms/ios/plist.d.ts +0 -1
  161. package/dist/src/platforms/ios/runner-client.d.ts +0 -38
  162. package/dist/src/platforms/ios/runner-errors.d.ts +0 -20
  163. package/dist/src/platforms/ios/runner-macos-products.d.ts +0 -3
  164. package/dist/src/platforms/ios/runner-session.d.ts +0 -30
  165. package/dist/src/platforms/ios/runner-transport.d.ts +0 -10
  166. package/dist/src/platforms/ios/runner-xctestrun-products.d.ts +0 -2
  167. package/dist/src/platforms/ios/runner-xctestrun.d.ts +0 -38
  168. package/dist/src/platforms/ios/screenshot-status-bar.d.ts +0 -2
  169. package/dist/src/platforms/ios/screenshot.d.ts +0 -14
  170. package/dist/src/platforms/ios/simctl.d.ts +0 -7
  171. package/dist/src/platforms/ios/simulator.d.ts +0 -11
  172. package/dist/src/platforms/permission-utils.d.ts +0 -9
  173. package/dist/src/recording/overlay.d.ts +0 -10
  174. package/dist/src/upload-client.d.ts +0 -7
  175. package/dist/src/utils/args.d.ts +0 -27
  176. package/dist/src/utils/cli-config.d.ts +0 -10
  177. package/dist/src/utils/cli-option-schema.d.ts +0 -19
  178. package/dist/src/utils/cli-options.d.ts +0 -13
  179. package/dist/src/utils/command-schema.d.ts +0 -123
  180. package/dist/src/utils/device-isolation.d.ts +0 -3
  181. package/dist/src/utils/device.d.ts +0 -35
  182. package/dist/src/utils/diagnostics.d.ts +0 -30
  183. package/dist/src/utils/errors.d.ts +0 -26
  184. package/dist/src/utils/exec.d.ts +0 -32
  185. package/dist/src/utils/finders.d.ts +0 -12
  186. package/dist/src/utils/interactors.d.ts +0 -38
  187. package/dist/src/utils/json-input.d.ts +0 -1
  188. package/dist/src/utils/keyed-lock.d.ts +0 -1
  189. package/dist/src/utils/output.d.ts +0 -27
  190. package/dist/src/utils/path-resolution.d.ts +0 -8
  191. package/dist/src/utils/payload-input.d.ts +0 -12
  192. package/dist/src/utils/process-identity.d.ts +0 -11
  193. package/dist/src/utils/remote-config.d.ts +0 -15
  194. package/dist/src/utils/remote-open.d.ts +0 -9
  195. package/dist/src/utils/retry.d.ts +0 -54
  196. package/dist/src/utils/screenshot-diff.d.ts +0 -23
  197. package/dist/src/utils/session-binding.d.ts +0 -18
  198. package/dist/src/utils/snapshot-lines.d.ts +0 -15
  199. package/dist/src/utils/snapshot.d.ts +0 -49
  200. package/dist/src/utils/text-surface.d.ts +0 -19
  201. package/dist/src/utils/timeouts.d.ts +0 -3
  202. package/dist/src/utils/version.d.ts +0 -2
  203. package/dist/src/utils/video.d.ts +0 -9
@@ -1,7 +1,16 @@
1
1
  import XCTest
2
2
 
3
3
  extension RunnerTests {
4
+ private static let collapsedTabCandidateTypes: Set<XCUIElement.ElementType> = [
5
+ .button,
6
+ .link,
7
+ .menuItem,
8
+ .other,
9
+ .staticText
10
+ ]
11
+
4
12
  private struct SnapshotTraversalContext {
13
+ let queryRoot: XCUIElement
5
14
  let rootSnapshot: XCUIElementSnapshot
6
15
  let viewport: CGRect
7
16
  let flatSnapshots: [XCUIElementSnapshot]
@@ -68,12 +77,34 @@ extension RunnerTests {
68
77
  return DataPayload(nodes: [], truncated: false)
69
78
  }
70
79
 
80
+ var cachedDescendantElements: [XCUIElement]?
81
+ func collapsedTabDescendants() -> [XCUIElement] {
82
+ if let cachedDescendantElements {
83
+ return cachedDescendantElements
84
+ }
85
+ let fetched = safeSnapshotElementsQuery {
86
+ context.queryRoot.descendants(matching: .any).allElementsBoundByIndex
87
+ }
88
+ cachedDescendantElements = fetched
89
+ return fetched
90
+ }
91
+
71
92
  var nodes: [SnapshotNode] = []
72
93
  var truncated = false
73
94
  let rootEvaluation = evaluateSnapshot(context.rootSnapshot, in: context)
74
95
  nodes.append(
75
96
  makeSnapshotNode(snapshot: context.rootSnapshot, evaluation: rootEvaluation, depth: 0, index: 0)
76
97
  )
98
+ if context.maxDepth > 0 {
99
+ let didTruncateFallback = appendCollapsedTabFallbackNodes(
100
+ to: &nodes,
101
+ containerSnapshot: context.rootSnapshot,
102
+ resolveElements: collapsedTabDescendants,
103
+ depth: 1,
104
+ nodeLimit: fastSnapshotLimit
105
+ )
106
+ truncated = truncated || didTruncateFallback
107
+ }
77
108
 
78
109
  var seen = Set<String>()
79
110
  var stack: [(XCUIElementSnapshot, Int, Int)] = context.rootSnapshot.children.map { ($0, 1, 1) }
@@ -119,6 +150,16 @@ extension RunnerTests {
119
150
  index: nodes.count
120
151
  )
121
152
  )
153
+ if visibleDepth < context.maxDepth {
154
+ let didTruncateFallback = appendCollapsedTabFallbackNodes(
155
+ to: &nodes,
156
+ containerSnapshot: snapshot,
157
+ resolveElements: collapsedTabDescendants,
158
+ depth: visibleDepth + 1,
159
+ nodeLimit: fastSnapshotLimit
160
+ )
161
+ truncated = truncated || didTruncateFallback
162
+ }
122
163
 
123
164
  }
124
165
 
@@ -247,6 +288,7 @@ extension RunnerTests {
247
288
 
248
289
  let (flatSnapshots, snapshotRanges) = flattenedSnapshots(rootSnapshot)
249
290
  return SnapshotTraversalContext(
291
+ queryRoot: queryRoot,
250
292
  rootSnapshot: rootSnapshot,
251
293
  viewport: viewport,
252
294
  flatSnapshots: flatSnapshots,
@@ -376,4 +418,183 @@ extension RunnerTests {
376
418
  if rect.isNull || rect.isEmpty { return false }
377
419
  return rect.intersects(viewport)
378
420
  }
421
+
422
+ private func appendCollapsedTabFallbackNodes(
423
+ to nodes: inout [SnapshotNode],
424
+ containerSnapshot: XCUIElementSnapshot,
425
+ resolveElements: () -> [XCUIElement],
426
+ depth: Int,
427
+ nodeLimit: Int
428
+ ) -> Bool {
429
+ let fallbackNodes = collapsedTabFallbackNodes(
430
+ for: containerSnapshot,
431
+ resolveElements: resolveElements,
432
+ startingIndex: nodes.count,
433
+ depth: depth
434
+ )
435
+ if fallbackNodes.isEmpty { return false }
436
+ let remaining = max(0, nodeLimit - nodes.count)
437
+ if remaining == 0 { return true }
438
+ nodes.append(contentsOf: fallbackNodes.prefix(remaining))
439
+ return fallbackNodes.count > remaining
440
+ }
441
+
442
+ private func collapsedTabFallbackNodes(
443
+ for containerSnapshot: XCUIElementSnapshot,
444
+ resolveElements: () -> [XCUIElement],
445
+ startingIndex: Int,
446
+ depth: Int
447
+ ) -> [SnapshotNode] {
448
+ if !containerSnapshot.children.isEmpty { return [] }
449
+ guard shouldExpandCollapsedTabContainer(containerSnapshot) else { return [] }
450
+ let containerFrame = containerSnapshot.frame
451
+ if containerFrame.isNull || containerFrame.isEmpty { return [] }
452
+
453
+ // Collapsed tab containers should be rare, so a full descendant scan is acceptable once per
454
+ // snapshot as a fallback for XCTest omitting the tab children from the snapshot tree.
455
+ let elements = resolveElements()
456
+ let candidates = elements.compactMap { element in
457
+ collapsedTabCandidateNode(
458
+ element: element,
459
+ containerSnapshot: containerSnapshot,
460
+ containerFrame: containerFrame
461
+ )
462
+ }
463
+ .sorted { left, right in
464
+ if left.rect.x != right.rect.x {
465
+ return left.rect.x < right.rect.x
466
+ }
467
+ return left.rect.y < right.rect.y
468
+ }
469
+
470
+ if candidates.count < 2 { return [] }
471
+ let rowMidpoints = candidates.map { $0.rect.y + ($0.rect.height / 2) }
472
+ let rowSpread = (rowMidpoints.max() ?? 0) - (rowMidpoints.min() ?? 0)
473
+ // Allow modest vertical jitter and short two-row wraps while still rejecting unrelated controls.
474
+ if rowSpread > max(24.0, Double(containerFrame.height) * 0.6) { return [] }
475
+
476
+ var seen = Set<String>()
477
+ let uniqueCandidates = candidates.filter { node in
478
+ let key = "\(node.type)-\(node.label ?? "")-\(node.identifier ?? "")-\(node.value ?? "")-\(node.rect.x)-\(node.rect.y)-\(node.rect.width)-\(node.rect.height)"
479
+ if seen.contains(key) { return false }
480
+ seen.insert(key)
481
+ return true
482
+ }
483
+ if uniqueCandidates.count < 2 { return [] }
484
+
485
+ return uniqueCandidates.enumerated().map { offset, node in
486
+ SnapshotNode(
487
+ index: startingIndex + offset,
488
+ type: node.type,
489
+ label: node.label,
490
+ identifier: node.identifier,
491
+ value: node.value,
492
+ rect: node.rect,
493
+ enabled: node.enabled,
494
+ hittable: node.hittable,
495
+ depth: depth
496
+ )
497
+ }
498
+ }
499
+
500
+ private func collapsedTabCandidateNode(
501
+ element: XCUIElement,
502
+ containerSnapshot: XCUIElementSnapshot,
503
+ containerFrame: CGRect
504
+ ) -> SnapshotNode? {
505
+ var node: SnapshotNode?
506
+ let exceptionMessage = RunnerObjCExceptionCatcher.catchException({
507
+ if !element.exists { return }
508
+ let elementType = element.elementType
509
+ if !Self.collapsedTabCandidateTypes.contains(elementType) { return }
510
+ let frame = element.frame
511
+ if frame.isNull || frame.isEmpty { return }
512
+ if frame.equalTo(containerFrame) { return }
513
+ let area = max(CGFloat(1), frame.width * frame.height)
514
+ let containerArea = max(CGFloat(1), containerFrame.width * containerFrame.height)
515
+ if area >= containerArea * 0.9 { return }
516
+ let center = CGPoint(x: frame.midX, y: frame.midY)
517
+ if !containerFrame.contains(center) { return }
518
+
519
+ let label = element.label.trimmingCharacters(in: .whitespacesAndNewlines)
520
+ let identifier = element.identifier.trimmingCharacters(in: .whitespacesAndNewlines)
521
+ let valueText = snapshotValueText(element)
522
+ let hasContent = !label.isEmpty || !identifier.isEmpty || valueText != nil
523
+ if !hasContent { return }
524
+ if sameSemanticElement(
525
+ containerSnapshot: containerSnapshot,
526
+ elementType: elementType,
527
+ label: label,
528
+ identifier: identifier
529
+ ) {
530
+ return
531
+ }
532
+
533
+ node = SnapshotNode(
534
+ index: 0,
535
+ type: elementTypeName(elementType),
536
+ label: label.isEmpty ? nil : label,
537
+ identifier: identifier.isEmpty ? nil : identifier,
538
+ value: valueText,
539
+ rect: snapshotRect(from: frame),
540
+ enabled: element.isEnabled,
541
+ hittable: element.isHittable,
542
+ depth: 0
543
+ )
544
+ })
545
+ if let exceptionMessage {
546
+ NSLog(
547
+ "AGENT_DEVICE_RUNNER_SNAPSHOT_TAB_FALLBACK_IGNORED_EXCEPTION=%@",
548
+ exceptionMessage
549
+ )
550
+ return nil
551
+ }
552
+ return node
553
+ }
554
+
555
+ private func shouldExpandCollapsedTabContainer(_ snapshot: XCUIElementSnapshot) -> Bool {
556
+ let frame = snapshot.frame
557
+ if frame.isNull || frame.isEmpty { return false }
558
+ if frame.width < max(CGFloat(160), frame.height * 1.75) { return false }
559
+ switch snapshot.elementType {
560
+ case .tabBar, .segmentedControl, .slider:
561
+ return true
562
+ default:
563
+ return false
564
+ }
565
+ }
566
+
567
+ private func snapshotValueText(_ element: XCUIElement) -> String? {
568
+ let text = String(describing: element.value ?? "")
569
+ .trimmingCharacters(in: .whitespacesAndNewlines)
570
+ return text.isEmpty ? nil : text
571
+ }
572
+
573
+ private func sameSemanticElement(
574
+ containerSnapshot: XCUIElementSnapshot,
575
+ elementType: XCUIElement.ElementType,
576
+ label: String,
577
+ identifier: String
578
+ ) -> Bool {
579
+ if containerSnapshot.elementType != elementType { return false }
580
+ let containerLabel = containerSnapshot.label.trimmingCharacters(in: .whitespacesAndNewlines)
581
+ let containerIdentifier = containerSnapshot.identifier
582
+ .trimmingCharacters(in: .whitespacesAndNewlines)
583
+ return containerLabel == label && containerIdentifier == identifier
584
+ }
585
+
586
+ private func safeSnapshotElementsQuery(_ fetch: () -> [XCUIElement]) -> [XCUIElement] {
587
+ var elements: [XCUIElement] = []
588
+ let exceptionMessage = RunnerObjCExceptionCatcher.catchException({
589
+ elements = fetch()
590
+ })
591
+ if let exceptionMessage {
592
+ NSLog(
593
+ "AGENT_DEVICE_RUNNER_SNAPSHOT_QUERY_IGNORED_EXCEPTION=%@",
594
+ exceptionMessage
595
+ )
596
+ return []
597
+ }
598
+ return elements
599
+ }
379
600
  }
@@ -1,16 +1,25 @@
1
1
  # agent-device iOS Runner
2
2
 
3
- This folder is reserved for the lightweight XCUITest runner used to provide element-level automation on iOS.
3
+ This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.
4
4
 
5
5
  ## Intent
6
+
6
7
  - Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
7
8
  - Allow local builds via `xcodebuild` and caching for faster subsequent runs.
8
9
  - Support simulator prebuilds where compatible.
9
10
 
10
11
  ## Status
11
- Planned for the automation layer. See `docs/ios-automation.md` and `docs/ios-runner-protocol.md`.
12
+
13
+ Current internal runner for iOS, tvOS, and macOS desktop automation.
14
+
15
+ Protocol and maintenance references:
16
+
17
+ - Protocol overview: [`RUNNER_PROTOCOL.md`](RUNNER_PROTOCOL.md)
18
+ - TypeScript client: [`../src/platforms/ios/runner-client.ts`](../src/platforms/ios/runner-client.ts)
19
+ - Swift wire models: [`AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift`](AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift)
12
20
 
13
21
  ## UITest Runner File Map
22
+
14
23
  `AgentDeviceRunnerUITests/RunnerTests` is split into focused files to reduce context size for contributors and LLM agents.
15
24
 
16
25
  - `RunnerTests.swift`: shared state/constants, `setUp()`, and `testCommand()` entry flow.
@@ -23,3 +32,9 @@ Planned for the automation layer. See `docs/ios-automation.md` and `docs/ios-run
23
32
  - `RunnerTests+Snapshot.swift`: fast/raw snapshot builders and include/filter helpers.
24
33
  - `RunnerTests+SystemModal.swift`: SpringBoard/system modal detection and modal snapshot shaping.
25
34
  - `RunnerTests+ScreenRecorder.swift`: nested `ScreenRecorder` implementation.
35
+
36
+ ## Protocol Notes
37
+
38
+ - The daemon posts JSON commands to `POST /command` on the runner's local HTTP listener.
39
+ - The runner responds with a JSON envelope shaped as `{ ok, data?, error? }`.
40
+ - The protocol is internal to `agent-device`; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.
@@ -0,0 +1,73 @@
1
+ # iOS Runner Protocol
2
+
3
+ The Apple runner speaks a small internal HTTP+JSON protocol between the TypeScript daemon and the XCUITest host. This protocol is a maintainer document, not part of the public user docs, but it should stay explicit so the TypeScript and Swift sides do not drift.
4
+
5
+ ## Transport
6
+
7
+ - Endpoint: `POST /command`
8
+ - Content type: `application/json`
9
+ - Request body: one JSON command object
10
+ - Response body: one JSON envelope
11
+
12
+ The daemon probes `http://127.0.0.1:<port>/command` for simulator and desktop flows, and can use a tunneled device address for physical iOS/tvOS devices before falling back to localhost.
13
+
14
+ ## Request Shape
15
+
16
+ Every request includes a `command` field. Additional fields depend on the command family.
17
+
18
+ Examples:
19
+
20
+ ```json
21
+ { "command": "tap", "x": 120, "y": 240 }
22
+ ```
23
+
24
+ ```json
25
+ {
26
+ "command": "snapshot",
27
+ "interactiveOnly": true,
28
+ "compact": true,
29
+ "depth": 2,
30
+ "scope": "app",
31
+ "raw": false
32
+ }
33
+ ```
34
+
35
+ ```json
36
+ { "command": "recordStart", "outPath": "/tmp/demo.mp4", "fps": 30 }
37
+ ```
38
+
39
+ The current command names are defined in:
40
+
41
+ - [`../src/platforms/ios/runner-client.ts`](../src/platforms/ios/runner-client.ts)
42
+ - [`AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift`](AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift)
43
+
44
+ ## Response Shape
45
+
46
+ Successful and failed responses use the same top-level envelope:
47
+
48
+ ```json
49
+ {
50
+ "ok": true,
51
+ "data": {
52
+ "message": "ok"
53
+ }
54
+ }
55
+ ```
56
+
57
+ ```json
58
+ {
59
+ "ok": false,
60
+ "error": {
61
+ "code": "UNSUPPORTED_OPERATION",
62
+ "message": "Unable to dismiss the iOS keyboard without a native dismiss gesture or control"
63
+ }
64
+ }
65
+ ```
66
+
67
+ `data` is command-specific. Common fields include snapshot nodes, text lookup results, gesture timing, visibility metadata, and screenshot or recording output details.
68
+
69
+ ## Maintenance Rules
70
+
71
+ - Treat the TypeScript and Swift wire models as a single contract.
72
+ - When adding, removing, or renaming a command, update the protocol fixtures/tests in the same change.
73
+ - Keep this file focused on the actual wire shape rather than implementation details of command execution.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "agent-device",
3
- "version": "0.10.2",
3
+ "version": "0.11.0",
4
4
  "description": "Unified control plane for physical and virtual devices via an agent-driven CLI.",
5
5
  "license": "MIT",
6
6
  "author": "Callstack",
@@ -31,13 +31,16 @@
31
31
  "build:all": "pnpm build:node && pnpm build:xcuitest",
32
32
  "ad": "node bin/agent-device.mjs",
33
33
  "format": "prettier --write src test skills",
34
- "prepublishOnly": "pnpm build:all",
35
34
  "prepack": "pnpm build:all",
36
35
  "typecheck": "tsc -p tsconfig.json",
37
- "test": "node --test",
38
- "test:unit": "node --test src/__tests__/*.test.ts src/core/__tests__/*.test.ts src/daemon/__tests__/*.test.ts src/daemon/handlers/__tests__/*.test.ts src/platforms/**/__tests__/*.test.ts src/utils/**/__tests__/*.test.ts",
36
+ "test": "vitest run",
37
+ "test:unit": "vitest run",
39
38
  "test:smoke": "node --test test/integration/smoke-*.test.ts",
40
- "test:integration": "node --test test/integration/*.test.ts"
39
+ "test:integration": "node --test test/integration/*.test.ts",
40
+ "test:replay:ios": "node --experimental-strip-types src/bin.ts test test/integration/replays/ios/simulator",
41
+ "test:replay:ios-device": "node --experimental-strip-types src/bin.ts test test/integration/replays/ios/device",
42
+ "test:replay:android": "node --experimental-strip-types src/bin.ts test test/integration/replays/android",
43
+ "test:replay:macos": "node --experimental-strip-types src/bin.ts test test/integration/replays/macos"
41
44
  },
42
45
  "files": [
43
46
  "bin",
@@ -73,10 +76,12 @@
73
76
  "pngjs": "^7.0.0"
74
77
  },
75
78
  "devDependencies": {
79
+ "@microsoft/api-extractor": "^7.52.10",
76
80
  "@rslib/core": "0.19.4",
77
81
  "@types/node": "^22.0.0",
78
82
  "@types/pngjs": "^6.0.5",
79
83
  "prettier": "^3.3.3",
80
- "typescript": "^5.9.3"
84
+ "typescript": "^5.9.3",
85
+ "vitest": "^4.1.2"
81
86
  }
82
87
  }
@@ -5,35 +5,61 @@ description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) a
5
5
 
6
6
  # agent-device
7
7
 
8
- Use this skill as a router.
8
+ Use this skill as a router with mandatory defaults. Read this file first. For normal device tasks, always load `references/bootstrap-install.md` and `references/exploration.md` before acting. Use bootstrap to confirm or establish deterministic setup. Use exploration for UI inspection, interaction, and verification once the app session is open.
9
+
10
+ ## Default operating rules
11
+
12
+ - Start conservative. Prefer read-only inspection before mutating the UI.
13
+ - Use plain `snapshot` when the task is to verify what text or structure is currently visible on screen.
14
+ - Use `snapshot -i` only when you need interactive refs such as `@e3` for a requested action or targeted query.
15
+ - Avoid speculative mutations. You may take the smallest reversible UI action needed to unblock inspection or complete the requested task, such as dismissing a popup, closing an alert, or clearing an unintended surface.
16
+ - Do not browse the web or use external sources unless the user explicitly asks.
17
+ - Re-snapshot after meaningful UI changes instead of reusing stale refs.
18
+ - Prefer `@ref` or selector targeting over raw coordinates.
19
+ - Ensure the correct target is pinned and an app session is open before interacting.
20
+ - Keep the loop short: `open` -> inspect/act -> verify if needed -> `close`.
21
+
22
+ ## Default flow
23
+
24
+ 1. Load [references/bootstrap-install.md](references/bootstrap-install.md) and [references/exploration.md](references/exploration.md) before acting on a normal device task.
25
+ 2. Use bootstrap first to confirm or establish the correct target, app install, and open app session.
26
+ 3. Once the app session is open and stable, use exploration for inspection, interaction, and verification.
27
+ 4. Start with plain `snapshot` if the goal is to read or verify what is visible.
28
+ 5. Escalate to `snapshot -i` only if you need refs for interactive exploration or a requested action.
29
+ 6. Use `get`, `is`, or `find` before mutating the UI when a read-only command can answer the question.
30
+ 7. End by capturing proof if needed, then `close`.
9
31
 
10
32
  ## QA modes
11
33
 
12
34
  - Open-ended bug hunt with reporting: use [../dogfood/SKILL.md](../dogfood/SKILL.md).
13
35
  - Pass/fail QA from acceptance criteria: stay in this skill, start with [references/bootstrap-install.md](references/bootstrap-install.md), then use the QA loop in [references/exploration.md](references/exploration.md).
14
36
 
15
- ## Mental model
37
+ ## Required references
16
38
 
17
- - First choose the correct target and open the app or session you want to work on.
18
- - Then inspect the current UI with `snapshot -i` and pick targets from the actual UI state.
19
- - Act with `press`, `fill`, `get`, `is`, `wait`, or `find`.
20
- - Re-snapshot after meaningful UI changes instead of reusing stale refs.
21
- - End by capturing proof if needed, then `close`.
39
+ - For every normal device task, after reading this file, load [references/bootstrap-install.md](references/bootstrap-install.md) first, then [references/exploration.md](references/exploration.md), before acting.
40
+ - Use bootstrap to confirm or establish deterministic setup, especially in sandbox or cloud environments.
41
+ - Use exploration once the app session is open and stable.
42
+ - Load additional references only when their scope is needed.
22
43
 
23
44
  ## Decision rules
24
45
 
25
46
  - Use plain `snapshot` when you need to verify whether text is visible.
26
47
  - Use `snapshot -i` mainly for interactive exploration and choosing refs.
48
+ - Use `get`, `is`, or `find` when they can answer the question without changing UI state.
27
49
  - Use `fill` to replace text.
28
50
  - Use `type` to append text.
51
+ - If the on-screen keyboard blocks the next step, prefer `keyboard dismiss` over navigation. On iOS, keep an app session open first; `keyboard status|get` remains Android-only.
52
+ - When a task asks to "go back", use plain `back` for predictable app-owned navigation and reserve `back --system` for platform back gestures or button semantics.
53
+ - Use `type --delay-ms` or `fill --delay-ms` for debounced search fields that drop characters when typed too quickly.
54
+ - If there is no simulator, no app install, or no open app session yet, switch to `bootstrap-install.md` instead of improvising setup steps.
55
+ - Use the smallest unblock action first when transient UI blocks inspection, but do not navigate, search, or enter new text just to make the UI reveal data unless the user asked for that interaction.
56
+ - Do not use external lookups to compensate for missing on-screen data unless the user asked for them.
57
+ - If the needed information is not exposed on screen, say that plainly instead of compensating with extra navigation, text entry, or web search.
29
58
  - Prefer `@ref` or selector targeting over raw coordinates.
30
- - Keep the default loop short: `open` -> explore/act -> optional debug or verify -> `close`.
31
59
 
32
- ## Choose a reference
60
+ ## Additional references
33
61
 
34
- - Pick target device, install, open, or manage sessions: [references/bootstrap-install.md](references/bootstrap-install.md)
35
- - Need to discover UI, pick refs, wait, query, or interact: [references/exploration.md](references/exploration.md)
36
62
  - Need logs, network, alerts, permissions, or failure triage: [references/debugging.md](references/debugging.md)
37
63
  - Need screenshots, diff, recording, replay maintenance, or perf data: [references/verification.md](references/verification.md)
38
64
  - Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: [references/macos-desktop.md](references/macos-desktop.md)
39
- - Need to connect to a remote `agent-device` daemon over HTTP or use tenant leases: [references/remote-tenancy.md](references/remote-tenancy.md)
65
+ - Need remote HTTP transport, `--remote-config` launches, or tenant leases on a remote macOS host: [references/remote-tenancy.md](references/remote-tenancy.md)
@@ -2,37 +2,79 @@
2
2
 
3
3
  ## When to open this file
4
4
 
5
- Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting.
5
+ Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting. This is the deterministic setup layer for sandbox, cloud, or other environments where install paths, device state, or app readiness may be uncertain.
6
6
 
7
- ## Main commands to reach for first
7
+ ## Open-first path
8
8
 
9
9
  - `devices`
10
+ - `apps`
10
11
  - `ensure-simulator`
11
12
  - `open`
12
- - `install` or `reinstall`
13
- - `close`
14
13
  - `session list`
15
14
 
15
+ ## Install path
16
+
17
+ - `install` or `reinstall`
18
+
16
19
  ## Most common mistake to avoid
17
20
 
18
21
  Do not start acting before you have pinned the correct target and opened an `app` session. In mixed-device environments, always pass `--device`, `--udid`, or `--serial`.
19
22
 
20
- ## Canonical loop
23
+ ## Deterministic setup rule
24
+
25
+ If there is no simulator, no app install, no open app session, or any uncertainty about where the app should come from, stay in this file and use deterministic setup commands or bootstrap scripts first. Do not improvise install paths or app-launch flows while exploring.
26
+
27
+ After setup is confirmed or completed, move to `exploration.md` before doing UI inspection or interaction.
28
+
29
+ ## Open-first rule
30
+
31
+ - If the user asks to test an app and does not provide an install artifact or explicit install instruction, try `open <app>` first.
32
+ - If `open <app>` fails, run `agent-device apps` and retry with a discovered app name before considering install steps.
33
+ - Do not install or reinstall on the first attempt unless the user explicitly asks for installation or provides a concrete artifact path or URL.
34
+ - When installation is required from a known location, prefer a checked-in shell script or other deterministic bootstrap command over ad hoc path guessing.
35
+
36
+ - If `open <app>` fails, or you are not sure which app name is available on the target, run `agent-device apps` first and choose from the discovered app list instead of guessing.
37
+ - Use `apps --platform <platform>` together with `--device`, `--udid`, or `--serial` when target selection matters.
38
+ - Once you have the correct app name, retry `open` with that exact discovered value.
39
+
40
+ ## Common starting points
41
+
42
+ These are examples, not required exact sequences. Use the smallest setup flow that matches the task.
43
+
44
+ ### Boot a simulator and open an app
21
45
 
22
46
  ```bash
23
47
  agent-device ensure-simulator --platform ios --device "iPhone 17 Pro" --boot
24
48
  agent-device open MyApp --platform ios --device "iPhone 17 Pro" --relaunch
25
- agent-device snapshot -i
26
- agent-device close
27
49
  ```
28
50
 
51
+ ### Install an app artifact
52
+
53
+ ```bash
54
+ agent-device install com.example.app ./build/app.apk --platform android --serial emulator-5554
55
+ ```
56
+
57
+ ```bash
58
+ agent-device install com.example.app ./build/MyApp.app --platform ios --device "iPhone 17 Pro"
59
+ ```
60
+
61
+ ## Install guidance
62
+
63
+ - Use `install <app> <path>` when the app may already be installed and you do not need a fresh-state reset.
64
+ - Use `reinstall <app> <path>` when you explicitly need uninstall plus install as one deterministic step.
65
+ - Keep install and open as separate phases. Do not turn them into one default command flow.
66
+ - Supported binary formats:
67
+ - Android: `.apk` and `.aab`
68
+ - iOS: `.app` and `.ipa`
69
+ - For iOS `.ipa` files, `<app>` is used as the bundle id or bundle name hint when the archive contains multiple app bundles.
70
+ - After install or reinstall, later use `open <app>` with the exact discovered or known package/bundle identifier, not the artifact path.
71
+
29
72
  ## Choose the right starting point
30
73
 
31
74
  - iOS local QA: prefer simulators unless the task explicitly requires physical hardware.
32
75
  - iOS in mixed simulator and device environments: run `ensure-simulator` first, then keep using `--device` or `--udid`.
33
76
  - TV targets: use `--target tv` together with `--platform` when the task is for tvOS or Android TV rather than phone or tablet surfaces.
34
77
  - Android binary flow: use `install` or `reinstall` for `.apk` or `.aab`, then open by installed package name.
35
- - Android React Native plus Metro flow: `reinstall <app> <apk>` first, then `open <package> --remote-config <path> --relaunch`.
36
78
  - macOS desktop app flow: use `open <app> --platform macos`. Only load [macos-desktop.md](macos-desktop.md) if a desktop surface or macOS-specific behavior matters.
37
79
 
38
80
  TV example:
@@ -95,8 +137,6 @@ export AGENT_DEVICE_PLATFORM=ios
95
137
  export AGENT_DEVICE_SESSION_LOCK=strip
96
138
 
97
139
  agent-device open MyApp --relaunch
98
- agent-device snapshot -i
99
- agent-device close
100
140
  ```
101
141
 
102
142
  - `AGENT_DEVICE_SESSION` plus `AGENT_DEVICE_PLATFORM` provides the default binding.
@@ -111,10 +151,7 @@ Android emulator variant:
111
151
  export AGENT_DEVICE_SESSION=qa-android
112
152
  export AGENT_DEVICE_PLATFORM=android
113
153
 
114
- agent-device reinstall MyApp /path/to/app-debug.apk --serial emulator-5554
115
154
  agent-device --session-lock reject open com.example.myapp --relaunch
116
- agent-device snapshot -i
117
- agent-device close --shutdown
118
155
  ```
119
156
 
120
157
  ## Scoped discovery
@@ -151,11 +188,14 @@ agent-device replay -u ./session.ad --session auth
151
188
  - Once the correct target and session are pinned, move to [exploration.md](exploration.md).
152
189
  - If opening, startup, permissions, or logs become the blocker, switch to [debugging.md](debugging.md).
153
190
 
154
- ## Install and open examples
191
+ ## Install examples
155
192
 
156
193
  ```bash
157
194
  agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554
158
- agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch
195
+ ```
196
+
197
+ ```bash
198
+ agent-device install com.example.app ./build/MyApp.ipa --platform ios --device "iPhone 17 Pro"
159
199
  ```
160
200
 
161
201
  Do not use `open <apk|aab> --relaunch` on Android.