mobile-debug-mcp 0.30.0 → 0.30.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/interact/index.js +330 -22
- package/dist/observe/index.js +1 -0
- package/dist/observe/snapshot-metadata.js +62 -1
- package/dist/server/tool-definitions.js +7 -2
- package/dist/server/tool-handlers.js +3 -1
- package/dist/server-core.js +1 -1
- package/docs/CHANGELOG.md +6 -0
- package/docs/rfcs/013-wait-and-synchronization-reliability.md +15 -35
- package/docs/rfcs/014-actionability-resolution.md +32 -30
- package/package.json +1 -1
- package/src/interact/index.ts +427 -33
- package/src/observe/index.ts +1 -0
- package/src/observe/snapshot-metadata.ts +69 -2
- package/src/server/tool-definitions.ts +7 -2
- package/src/server/tool-handlers.ts +3 -1
- package/src/server-core.ts +1 -1
- package/src/types.ts +24 -0
- package/test/unit/interact/adjust_control.test.ts +104 -0
- package/test/unit/interact/subtree_collection.test.ts +24 -0
- package/test/unit/interact/tap_element.test.ts +71 -0
- package/test/unit/interact/wait_for_ui_change.test.ts +92 -1
- package/test/unit/observe/snapshot_metadata.test.ts +67 -0
|
@@ -283,26 +283,13 @@ wait_for_ui_change({
|
|
|
283
283
|
deviceId?: string,
|
|
284
284
|
timeout_ms?: number,
|
|
285
285
|
stability_window_ms?: number,
|
|
286
|
-
expected_change?: string
|
|
286
|
+
expected_change?: string,
|
|
287
|
+
scope?: "screen" | "subtree",
|
|
288
|
+
target?: string
|
|
287
289
|
})
|
|
288
290
|
```
|
|
289
291
|
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
The following fields are proposed extensions and are not currently part of the shipped contract:
|
|
293
|
-
|
|
294
|
-
```json
|
|
295
|
-
{
|
|
296
|
-
"scope": "screen" | "subtree",
|
|
297
|
-
"target": "element_id"
|
|
298
|
-
}
|
|
299
|
-
```
|
|
300
|
-
|
|
301
|
-
These extensions are illustrative and intended to support:
|
|
302
|
-
|
|
303
|
-
- subtree synchronization
|
|
304
|
-
- scoped convergence
|
|
305
|
-
- token-efficient synchronization
|
|
292
|
+
`scope=subtree` requires a target `element_id` and narrows synchronization to the resolved element subtree.
|
|
306
293
|
|
|
307
294
|
### Semantics
|
|
308
295
|
|
|
@@ -512,17 +499,12 @@ The current implementation exposes:
|
|
|
512
499
|
- `snapshot_revision`
|
|
513
500
|
- `captured_at_ms`
|
|
514
501
|
- advisory `loading_state`
|
|
502
|
+
- `scope` and `target` on wait responses
|
|
503
|
+
- `stability_state` and scoped `change_summary` on wait responses
|
|
515
504
|
|
|
516
505
|
The `loading_state` field is currently an advisory structured object rather than a simple enum value.
|
|
517
506
|
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
- `fresh`
|
|
521
|
-
- `stale`
|
|
522
|
-
- `transient`
|
|
523
|
-
- `stable`
|
|
524
|
-
|
|
525
|
-
Those concepts in this RFC describe synchronization semantics rather than currently shipped payload fields.
|
|
507
|
+
Wait responses also include a `snapshot_freshness_ms` age value so callers can reason about freshness without inferring it from revision numbers alone.
|
|
526
508
|
|
|
527
509
|
### Requirements
|
|
528
510
|
|
|
@@ -533,15 +515,7 @@ Those concepts in this RFC describe synchronization semantics rather than curren
|
|
|
533
515
|
|
|
534
516
|
### Proposed Stability States
|
|
535
517
|
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
Potential states include:
|
|
539
|
-
|
|
540
|
-
- `transient`
|
|
541
|
-
- `stabilizing`
|
|
542
|
-
- `stable`
|
|
543
|
-
|
|
544
|
-
These are not currently emitted by the shipped implementation.
|
|
518
|
+
The shipped implementation emits `transient` while a wait is still converging and `stable` once the stabilization window has completed.
|
|
545
519
|
|
|
546
520
|
---
|
|
547
521
|
|
|
@@ -586,6 +560,8 @@ Examples:
|
|
|
586
560
|
- improve synchronization latency
|
|
587
561
|
- improve Compose hierarchy handling
|
|
588
562
|
|
|
563
|
+
The shipped runtime exposes scope-aware waits and snapshot delta summaries on UI tree and debug snapshot responses to keep this signal lightweight.
|
|
564
|
+
|
|
589
565
|
---
|
|
590
566
|
|
|
591
567
|
## 7.5 Incremental Snapshot Delivery
|
|
@@ -599,6 +575,8 @@ Example capabilities:
|
|
|
599
575
|
- partial hierarchy refreshes
|
|
600
576
|
- revision patch delivery
|
|
601
577
|
|
|
578
|
+
The shipped runtime currently exposes revision-aware `snapshot_delta` metadata rather than transport-level patch payloads.
|
|
579
|
+
|
|
602
580
|
### Dependency
|
|
603
581
|
|
|
604
582
|
Incremental synchronization depends on stable element identity semantics.
|
|
@@ -826,6 +804,8 @@ Introduce:
|
|
|
826
804
|
- diff-based synchronization
|
|
827
805
|
- subtree invalidation
|
|
828
806
|
|
|
807
|
+
The runtime now exposes scoped diff summaries for waits, but full partial-hierarchy patch delivery remains a future protocol extension.
|
|
808
|
+
|
|
829
809
|
---
|
|
830
810
|
|
|
831
811
|
## Phase 4
|
|
@@ -887,4 +867,4 @@ This RFC introduces explicit synchronization semantics that:
|
|
|
887
867
|
- establish consistent sequencing expectations
|
|
888
868
|
- create a foundation for incremental synchronization capabilities
|
|
889
869
|
|
|
890
|
-
These changes are intended to improve reliability, reduce retries, and make agent behavior substantially more predictable across asynchronous mobile UI frameworks.
|
|
870
|
+
These changes are intended to improve reliability, reduce retries, and make agent behavior substantially more predictable across asynchronous mobile UI frameworks.
|
|
@@ -34,7 +34,7 @@ RFC 014 MUST be applied AFTER RFC 005/007 resolution and BEFORE any interaction
|
|
|
34
34
|
|
|
35
35
|
---
|
|
36
36
|
|
|
37
|
-
#
|
|
37
|
+
# 3. Problem Statement
|
|
38
38
|
|
|
39
39
|
Current interaction failures arise when actions are attempted on elements that are:
|
|
40
40
|
|
|
@@ -54,7 +54,7 @@ There is currently no single authoritative model defining when an element is saf
|
|
|
54
54
|
|
|
55
55
|
---
|
|
56
56
|
|
|
57
|
-
#
|
|
57
|
+
# 4. Goals
|
|
58
58
|
|
|
59
59
|
The system should:
|
|
60
60
|
|
|
@@ -66,7 +66,7 @@ The system should:
|
|
|
66
66
|
|
|
67
67
|
---
|
|
68
68
|
|
|
69
|
-
#
|
|
69
|
+
# 5. Non-Goals
|
|
70
70
|
|
|
71
71
|
This RFC does NOT:
|
|
72
72
|
|
|
@@ -78,11 +78,11 @@ This RFC does NOT:
|
|
|
78
78
|
|
|
79
79
|
---
|
|
80
80
|
|
|
81
|
-
#
|
|
81
|
+
# 6. Definition: Actionability
|
|
82
82
|
|
|
83
83
|
An element is considered **actionable** if ALL of the following conditions are satisfied:
|
|
84
84
|
|
|
85
|
-
##
|
|
85
|
+
## 6.1 Structural Validity
|
|
86
86
|
|
|
87
87
|
- Element exists in the current UI hierarchy snapshot
|
|
88
88
|
- Element identity is stable (per Richer Element Identity model)
|
|
@@ -97,23 +97,23 @@ Element identity MUST be derived using the following priority order:
|
|
|
97
97
|
|
|
98
98
|
If identity cannot be resolved consistently across snapshots, the element MUST be treated as non-actionable due to instability.
|
|
99
99
|
|
|
100
|
-
##
|
|
100
|
+
## 6.2 Visibility
|
|
101
101
|
|
|
102
102
|
- Element is visible within the current viewport OR scrollable container
|
|
103
103
|
- Element is not explicitly hidden (e.g. visibility = gone / hidden)
|
|
104
104
|
|
|
105
|
-
##
|
|
105
|
+
## 6.3 Enabled State
|
|
106
106
|
|
|
107
107
|
- Element is not disabled
|
|
108
108
|
- Element accepts interaction for the requested action type
|
|
109
|
-
- Must be interpreted relative to action_type (see Section
|
|
109
|
+
- Must be interpreted relative to action_type (see Section 7.4)
|
|
110
110
|
|
|
111
|
-
##
|
|
111
|
+
## 6.4 Stability Requirement
|
|
112
112
|
|
|
113
113
|
- UI must be in a "stable" state per RFC 013 stabilization rules
|
|
114
114
|
- No synchronization-relevant mutation is currently active
|
|
115
115
|
|
|
116
|
-
##
|
|
116
|
+
## 6.5 Interaction Eligibility
|
|
117
117
|
|
|
118
118
|
- Element is not in a transitional animation state that blocks interaction
|
|
119
119
|
- Element is not temporarily detached or reparenting in the hierarchy
|
|
@@ -122,11 +122,11 @@ If identity cannot be resolved consistently across snapshots, the element MUST b
|
|
|
122
122
|
|
|
123
123
|
---
|
|
124
124
|
|
|
125
|
-
#
|
|
125
|
+
# 7. Actionability Evaluation Model (Runtime Contract)
|
|
126
126
|
|
|
127
127
|
Actionability is evaluated using runtime-provided fields from the resolved element and snapshot system.
|
|
128
128
|
|
|
129
|
-
##
|
|
129
|
+
## 7.1 Predicate Definition
|
|
130
130
|
|
|
131
131
|
```text
|
|
132
132
|
is_actionable(element, snapshot, normalized_action_type) =
|
|
@@ -139,7 +139,7 @@ is_actionable(element, snapshot, normalized_action_type) =
|
|
|
139
139
|
AND identity_is_stable(element)
|
|
140
140
|
```
|
|
141
141
|
|
|
142
|
-
##
|
|
142
|
+
## 7.2 Field Mapping
|
|
143
143
|
|
|
144
144
|
| Predicate Component | Source Field |
|
|
145
145
|
|--------------------|-------------|
|
|
@@ -150,13 +150,13 @@ is_actionable(element, snapshot, normalized_action_type) =
|
|
|
150
150
|
| actionability_by_type(action_type, element) | derived runtime interaction capability from visible, enabled, clickable proxy signals (no hit-test dependency) |
|
|
151
151
|
| identity_is_stable | stable_id or fallback identity resolution |
|
|
152
152
|
|
|
153
|
-
##
|
|
153
|
+
## 7.3 Identity Stability Rule
|
|
154
154
|
|
|
155
155
|
Identity is considered stable if `stable_id` remains unchanged across the latest two snapshots.
|
|
156
156
|
|
|
157
157
|
If identity changes, the element MUST be re-resolved before actionability can be evaluated.
|
|
158
158
|
|
|
159
|
-
##
|
|
159
|
+
## 7.4 Action Type Applicability (Core Contract)
|
|
160
160
|
|
|
161
161
|
Action type is a PRIMARY dimension of actionability and MUST be evaluated explicitly for every interaction.
|
|
162
162
|
|
|
@@ -202,7 +202,7 @@ No constraint MAY override another constraint. No priority ordering between fiel
|
|
|
202
202
|
|
|
203
203
|
Evaluation MUST produce identical results for identical inputs (element, snapshot, action_type).
|
|
204
204
|
|
|
205
|
-
##
|
|
205
|
+
## 7.5 Deterministic Evaluation Guarantees
|
|
206
206
|
|
|
207
207
|
Actionability evaluation is a pure function of:
|
|
208
208
|
|
|
@@ -222,7 +222,7 @@ Short-circuit evaluation is permitted for performance but MUST NOT change final
|
|
|
222
222
|
|
|
223
223
|
---
|
|
224
224
|
|
|
225
|
-
#
|
|
225
|
+
# 8. Resolution Lifecycle
|
|
226
226
|
|
|
227
227
|
Actionability MUST be evaluated in the following sequence:
|
|
228
228
|
|
|
@@ -238,7 +238,7 @@ If any step fails, the element MUST be considered non-actionable.
|
|
|
238
238
|
|
|
239
239
|
---
|
|
240
240
|
|
|
241
|
-
#
|
|
241
|
+
# 9. Interaction Guardrail
|
|
242
242
|
|
|
243
243
|
Actions MUST NOT be executed unless actionability resolution returns true.
|
|
244
244
|
|
|
@@ -250,9 +250,9 @@ If actionability is false:
|
|
|
250
250
|
|
|
251
251
|
---
|
|
252
252
|
|
|
253
|
-
#
|
|
253
|
+
# 10. Failure Modes
|
|
254
254
|
|
|
255
|
-
##
|
|
255
|
+
## 10.1 Transient Non-Actionability
|
|
256
256
|
|
|
257
257
|
Caused by:
|
|
258
258
|
|
|
@@ -267,7 +267,7 @@ Mitigation:
|
|
|
267
267
|
|
|
268
268
|
---
|
|
269
269
|
|
|
270
|
-
##
|
|
270
|
+
## 10.2 Stale Snapshot
|
|
271
271
|
|
|
272
272
|
Caused by:
|
|
273
273
|
|
|
@@ -280,7 +280,7 @@ Mitigation:
|
|
|
280
280
|
|
|
281
281
|
---
|
|
282
282
|
|
|
283
|
-
##
|
|
283
|
+
## 10.3 False Positives (appears actionable but is not)
|
|
284
284
|
|
|
285
285
|
Caused by:
|
|
286
286
|
|
|
@@ -293,7 +293,7 @@ Mitigation:
|
|
|
293
293
|
|
|
294
294
|
---
|
|
295
295
|
|
|
296
|
-
#
|
|
296
|
+
# 11. Platform Considerations
|
|
297
297
|
|
|
298
298
|
## Android
|
|
299
299
|
|
|
@@ -307,7 +307,7 @@ Mitigation:
|
|
|
307
307
|
|
|
308
308
|
---
|
|
309
309
|
|
|
310
|
-
#
|
|
310
|
+
# 12. Integration with RFC 013
|
|
311
311
|
|
|
312
312
|
Actionability Resolution depends on:
|
|
313
313
|
|
|
@@ -318,7 +318,7 @@ It MUST NOT evaluate elements from unstable snapshots.
|
|
|
318
318
|
|
|
319
319
|
---
|
|
320
320
|
|
|
321
|
-
#
|
|
321
|
+
# 13. Telemetry
|
|
322
322
|
|
|
323
323
|
Systems SHOULD track:
|
|
324
324
|
|
|
@@ -329,7 +329,7 @@ Systems SHOULD track:
|
|
|
329
329
|
|
|
330
330
|
---
|
|
331
331
|
|
|
332
|
-
#
|
|
332
|
+
# 14. Rollout Strategy
|
|
333
333
|
|
|
334
334
|
## Phase 1
|
|
335
335
|
|
|
@@ -343,9 +343,11 @@ Systems SHOULD track:
|
|
|
343
343
|
|
|
344
344
|
- Platform-specific interaction heuristics
|
|
345
345
|
|
|
346
|
+
The shipped interaction layer now applies iOS semantic tap eligibility in addition to proxy clickability checks.
|
|
347
|
+
|
|
346
348
|
---
|
|
347
349
|
|
|
348
|
-
#
|
|
350
|
+
# 15. Dependencies
|
|
349
351
|
|
|
350
352
|
## Depends On
|
|
351
353
|
|
|
@@ -355,7 +357,7 @@ Systems SHOULD track:
|
|
|
355
357
|
|
|
356
358
|
---
|
|
357
359
|
|
|
358
|
-
#
|
|
360
|
+
# 15.1 Acceptance Criteria
|
|
359
361
|
|
|
360
362
|
An implementation of RFC 014 is considered correct if:
|
|
361
363
|
|
|
@@ -366,7 +368,7 @@ An implementation of RFC 014 is considered correct if:
|
|
|
366
368
|
- All failures produce a defined failure code
|
|
367
369
|
- Actionability is only evaluated AFTER RFC 005/007 resolution
|
|
368
370
|
|
|
369
|
-
#
|
|
371
|
+
# 16. Failure Codes (Actionability Gate)
|
|
370
372
|
|
|
371
373
|
## Failure Code Derivation Matrix
|
|
372
374
|
|
|
@@ -385,7 +387,7 @@ These are the only supported actionability failure codes.
|
|
|
385
387
|
|
|
386
388
|
Implementations SHOULD return the structured code together with a human-readable reason.
|
|
387
389
|
|
|
388
|
-
#
|
|
390
|
+
# 17. Conclusion
|
|
389
391
|
|
|
390
392
|
Actionability Resolution provides the final gate before UI interaction execution.
|
|
391
393
|
|