mobile-debug-mcp 0.28.0 → 0.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +13 -0
- package/README.md +44 -21
- package/dist/interact/index.js +30 -25
- package/dist/server/tool-definitions.js +1 -1
- package/dist/server/tool-handlers.js +1 -1
- package/dist/server-core.js +12 -2
- package/dist/server.js +5 -3
- package/docs/CHANGELOG.md +9 -0
- package/docs/ROADMAP.md +36 -28
- package/docs/rfcs/013-wait-and-synchronization-reliability.md +890 -0
- package/docs/rfcs/014-actionability-resolution.md +392 -0
- package/docs/specs/mcp-tooling-spec-v1.md +28 -0
- package/docs/tools/interact.md +6 -0
- package/package.json +1 -1
- package/src/interact/index.ts +29 -24
- package/src/server/tool-definitions.ts +1 -1
- package/src/server/tool-handlers.ts +1 -1
- package/src/server-core.ts +17 -1
- package/src/server.ts +5 -3
- package/test/unit/interact/wait_for_ui_change.test.ts +105 -52
package/AGENTS.md
CHANGED
|
@@ -75,3 +75,16 @@ For test authoring details, rely on the `test-authoring` skill package rather th
|
|
|
75
75
|
## Notes for maintainers
|
|
76
76
|
|
|
77
77
|
This file is intentionally short. Keep task-specific guidance in `skills/...` so multiple agent systems can reuse the same instructions.
|
|
78
|
+
|
|
79
|
+
## Testing
|
|
80
|
+
|
|
81
|
+
- `npm run test:unit` runs every automated unit test under `test/unit/...`
|
|
82
|
+
- `npm run test:device` runs the automated device smoke checks under `test/device/automated/...`
|
|
83
|
+
- `npm run verify` runs the default maintainer verification sequence: lint, build, and unit tests
|
|
84
|
+
- Manual and debug-oriented device scripts live under `test/device/manual/...` and are not part of the default test commands
|
|
85
|
+
|
|
86
|
+
## Utility Scripts
|
|
87
|
+
|
|
88
|
+
- `npm run healthcheck` runs the `idb`/tooling healthcheck helper from `src/utils/cli/idb/check-idb.ts`
|
|
89
|
+
- `npm run install-idb` runs the guided `idb` installer helper from `src/utils/cli/idb/install-idb.ts`
|
|
90
|
+
- `npm run preflight-ios` runs the iOS preflight helper from `src/utils/cli/ios/preflight-ios.ts`
|
package/README.md
CHANGED
|
@@ -16,7 +16,11 @@ A minimal, secure MCP server for AI-assisted mobile development. Build, install,
|
|
|
16
16
|
- Xcode command-line tools for iOS support
|
|
17
17
|
- [idb](https://github.com/facebook/idb) for iOS device support
|
|
18
18
|
|
|
19
|
-
## Configuration
|
|
19
|
+
## Configuration
|
|
20
|
+
|
|
21
|
+
<details>
|
|
22
|
+
|
|
23
|
+
<summary>Android Studio</summary>
|
|
20
24
|
|
|
21
25
|
```json
|
|
22
26
|
{
|
|
@@ -29,7 +33,45 @@ A minimal, secure MCP server for AI-assisted mobile development. Build, install,
|
|
|
29
33
|
}
|
|
30
34
|
}
|
|
31
35
|
```
|
|
32
|
-
|
|
36
|
+
|
|
37
|
+
</details>
|
|
38
|
+
|
|
39
|
+
<details>
|
|
40
|
+
|
|
41
|
+
<summary>Copilot</summary>
|
|
42
|
+
|
|
43
|
+
```json
|
|
44
|
+
{
|
|
45
|
+
"mcpServers": {
|
|
46
|
+
"mobile-debug": {
|
|
47
|
+
"command": "npx",
|
|
48
|
+
"args": ["--yes","mobile-debug-mcp","server"],
|
|
49
|
+
"env": { "ADB_PATH": "/path/to/adb", "XCRUN_PATH": "/usr/bin/xcrun", "IDB_PATH": "/path/to/idb" }
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
</details>
|
|
56
|
+
|
|
57
|
+
<details>
|
|
58
|
+
|
|
59
|
+
<summary>Codex</summary>
|
|
60
|
+
|
|
61
|
+
Use STDIO
|
|
62
|
+
|
|
63
|
+
command: npx
|
|
64
|
+
|
|
65
|
+
args:
|
|
66
|
+
* --yes
|
|
67
|
+
* mobile-debug-mcp
|
|
68
|
+
|
|
69
|
+
environment variables:
|
|
70
|
+
* ADB_PATH: /path/to/adb
|
|
71
|
+
* XCRUN_PATH: /usr/bin/xcrun
|
|
72
|
+
* IDC_PATH: /path/to/idb"
|
|
73
|
+
|
|
74
|
+
</details>
|
|
33
75
|
|
|
34
76
|
## Usage
|
|
35
77
|
|
|
@@ -48,25 +90,6 @@ Feature building:
|
|
|
48
90
|
- Agents: [AGENTS.md](AGENTS.md) — cold-start guidance for autonomous agents entering the public repo
|
|
49
91
|
- Skills: [skills/README.md](skills/README.md) — portable Markdown skill packages for agents such as Copilot, Codex, Claude, or custom systems
|
|
50
92
|
|
|
51
|
-
## Testing
|
|
52
|
-
|
|
53
|
-
- `npm run test:unit` runs every automated unit test under `test/unit/...`
|
|
54
|
-
- `npm run test:device` runs the automated device smoke checks under `test/device/automated/...`
|
|
55
|
-
- `npm run verify` runs the default maintainer verification sequence: lint, build, and unit tests
|
|
56
|
-
- Manual and debug-oriented device scripts live under `test/device/manual/...` and are not part of the default test commands
|
|
57
|
-
|
|
58
|
-
## Utility Scripts
|
|
59
|
-
|
|
60
|
-
- `npm run healthcheck` runs the `idb`/tooling healthcheck helper from `src/utils/cli/idb/check-idb.ts`
|
|
61
|
-
- `npm run install-idb` runs the guided `idb` installer helper from `src/utils/cli/idb/install-idb.ts`
|
|
62
|
-
- `npm run preflight-ios` runs the iOS preflight helper from `src/utils/cli/ios/preflight-ios.ts`
|
|
63
|
-
|
|
64
|
-
## Agent skills
|
|
65
|
-
|
|
66
|
-
- `skills/mcp-builder/` contains reusable build/install guidance for agents
|
|
67
|
-
- `skills/test-authoring/` contains reusable test-creation guidance aligned to this repo's current test structure
|
|
68
|
-
- Skills are written as plain Markdown packages so they can be consumed by different agent systems rather than one vendor-specific runtime
|
|
69
|
-
|
|
70
93
|
## License
|
|
71
94
|
|
|
72
95
|
MIT
|
package/dist/interact/index.js
CHANGED
|
@@ -205,6 +205,9 @@ export class ToolsInteract {
|
|
|
205
205
|
}
|
|
206
206
|
return null;
|
|
207
207
|
}
|
|
208
|
+
static _uiChangeSignaturesEqual(left, right) {
|
|
209
|
+
return left.hierarchy === right.hierarchy && left.text === right.text && left.state === right.state;
|
|
210
|
+
}
|
|
208
211
|
static _resolvedTargetFromElement(elementId, element, index) {
|
|
209
212
|
return {
|
|
210
213
|
elementId,
|
|
@@ -1708,13 +1711,16 @@ export class ToolsInteract {
|
|
|
1708
1711
|
}
|
|
1709
1712
|
};
|
|
1710
1713
|
}
|
|
1711
|
-
static async waitForUIChangeHandler({ platform, deviceId, timeout_ms = 60000, stability_window_ms =
|
|
1714
|
+
static async waitForUIChangeHandler({ platform, deviceId, timeout_ms = 60000, stability_window_ms = 300, expected_change }) {
|
|
1712
1715
|
const start = Date.now();
|
|
1713
1716
|
const pollIntervalMs = 300;
|
|
1714
|
-
const stabilityWindow = Math.max(0, typeof stability_window_ms === 'number' ? stability_window_ms :
|
|
1717
|
+
const stabilityWindow = Math.max(0, typeof stability_window_ms === 'number' ? stability_window_ms : 300);
|
|
1715
1718
|
let baseline = null;
|
|
1716
1719
|
let lastObservedRevision = null;
|
|
1717
1720
|
let lastLoadingState = null;
|
|
1721
|
+
let candidateSignatures = null;
|
|
1722
|
+
let candidateObservedChange = null;
|
|
1723
|
+
let candidateSinceMs = null;
|
|
1718
1724
|
while (Date.now() - start < timeout_ms) {
|
|
1719
1725
|
try {
|
|
1720
1726
|
const tree = await ToolsObserve.getUITreeHandler({ platform, deviceId });
|
|
@@ -1727,30 +1733,29 @@ export class ToolsInteract {
|
|
|
1727
1733
|
else {
|
|
1728
1734
|
const observedChange = ToolsInteract._matchesUiChange(expected_change, baseline, signatures);
|
|
1729
1735
|
if (observedChange) {
|
|
1730
|
-
if (
|
|
1731
|
-
|
|
1732
|
-
|
|
1733
|
-
|
|
1734
|
-
const confirmChange = ToolsInteract._matchesUiChange(expected_change, baseline, confirmSignatures);
|
|
1735
|
-
if (!confirmChange || confirmSignatures.hierarchy !== signatures.hierarchy || confirmSignatures.text !== signatures.text || confirmSignatures.state !== signatures.state) {
|
|
1736
|
-
lastObservedRevision = typeof confirmTree?.snapshot_revision === 'number' ? confirmTree.snapshot_revision : lastObservedRevision;
|
|
1737
|
-
lastLoadingState = confirmTree?.loading_state ?? lastLoadingState;
|
|
1738
|
-
await new Promise(resolve => setTimeout(resolve, pollIntervalMs));
|
|
1739
|
-
continue;
|
|
1740
|
-
}
|
|
1741
|
-
lastObservedRevision = typeof confirmTree?.snapshot_revision === 'number' ? confirmTree.snapshot_revision : lastObservedRevision;
|
|
1742
|
-
lastLoadingState = confirmTree?.loading_state ?? lastLoadingState;
|
|
1736
|
+
if (!candidateSignatures || !ToolsInteract._uiChangeSignaturesEqual(candidateSignatures, signatures) || candidateObservedChange !== observedChange) {
|
|
1737
|
+
candidateSignatures = signatures;
|
|
1738
|
+
candidateObservedChange = observedChange;
|
|
1739
|
+
candidateSinceMs = Date.now();
|
|
1743
1740
|
}
|
|
1744
|
-
|
|
1745
|
-
|
|
1746
|
-
|
|
1747
|
-
|
|
1748
|
-
|
|
1749
|
-
|
|
1750
|
-
|
|
1751
|
-
|
|
1752
|
-
|
|
1753
|
-
|
|
1741
|
+
const stableForMs = candidateSinceMs === null ? 0 : Date.now() - candidateSinceMs;
|
|
1742
|
+
if (stabilityWindow === 0 || stableForMs >= stabilityWindow) {
|
|
1743
|
+
return {
|
|
1744
|
+
success: true,
|
|
1745
|
+
observed_change: candidateObservedChange ?? observedChange,
|
|
1746
|
+
snapshot_revision: lastObservedRevision ?? undefined,
|
|
1747
|
+
timeout: false,
|
|
1748
|
+
elapsed_ms: Date.now() - start,
|
|
1749
|
+
expected_change,
|
|
1750
|
+
loading_state: lastLoadingState ?? null,
|
|
1751
|
+
reason: 'UI change observed'
|
|
1752
|
+
};
|
|
1753
|
+
}
|
|
1754
|
+
}
|
|
1755
|
+
else {
|
|
1756
|
+
candidateSignatures = null;
|
|
1757
|
+
candidateObservedChange = null;
|
|
1758
|
+
candidateSinceMs = null;
|
|
1754
1759
|
}
|
|
1755
1760
|
}
|
|
1756
1761
|
}
|
|
@@ -391,7 +391,7 @@ Failure Handling:
|
|
|
391
391
|
deviceId: { type: 'string', description: 'Optional device id/udid to target' },
|
|
392
392
|
expected_change: { type: 'string', enum: ['hierarchy_diff', 'text_change', 'state_change'], description: 'Optional type of UI change to wait for' },
|
|
393
393
|
timeout_ms: { type: 'number', description: 'Timeout in ms to wait for change (default 60000)', default: 60000 },
|
|
394
|
-
stability_window_ms: { type: 'number', description: 'How long the change must remain stable before success (default
|
|
394
|
+
stability_window_ms: { type: 'number', description: 'How long the change must remain stable before success (default 300)', default: 300 }
|
|
395
395
|
}
|
|
396
396
|
}
|
|
397
397
|
},
|
|
@@ -266,7 +266,7 @@ async function handleWaitForUIChange(args) {
|
|
|
266
266
|
const platform = getStringArg(args, 'platform');
|
|
267
267
|
const deviceId = getStringArg(args, 'deviceId');
|
|
268
268
|
const timeout_ms = getNumberArg(args, 'timeout_ms') ?? 60000;
|
|
269
|
-
const stability_window_ms = getNumberArg(args, 'stability_window_ms') ??
|
|
269
|
+
const stability_window_ms = getNumberArg(args, 'stability_window_ms') ?? 300;
|
|
270
270
|
const expected_change = getStringArg(args, 'expected_change');
|
|
271
271
|
const res = await ToolsInteract.waitForUIChangeHandler({ platform, deviceId, timeout_ms, stability_window_ms, expected_change });
|
|
272
272
|
return wrapResponse(res);
|
package/dist/server-core.js
CHANGED
|
@@ -1,19 +1,29 @@
|
|
|
1
1
|
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
|
|
2
|
-
import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
|
|
2
|
+
import { ListResourcesRequestSchema, ListResourceTemplatesRequestSchema, ReadResourceRequestSchema, ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
|
|
3
3
|
import { wrapResponse } from './server/common.js';
|
|
4
4
|
import { toolDefinitions } from './server/tool-definitions.js';
|
|
5
5
|
import { handleToolCall } from './server/tool-handlers.js';
|
|
6
6
|
export { wrapResponse, toolDefinitions, handleToolCall };
|
|
7
7
|
export const serverInfo = {
|
|
8
8
|
name: 'mobile-debug-mcp',
|
|
9
|
-
version: '0.
|
|
9
|
+
version: '0.30.0'
|
|
10
10
|
};
|
|
11
11
|
export function createServer() {
|
|
12
12
|
const server = new Server(serverInfo, {
|
|
13
13
|
capabilities: {
|
|
14
|
+
resources: {},
|
|
14
15
|
tools: {}
|
|
15
16
|
}
|
|
16
17
|
});
|
|
18
|
+
server.setRequestHandler(ListResourcesRequestSchema, async () => ({
|
|
19
|
+
resources: []
|
|
20
|
+
}));
|
|
21
|
+
server.setRequestHandler(ListResourceTemplatesRequestSchema, async () => ({
|
|
22
|
+
resourceTemplates: []
|
|
23
|
+
}));
|
|
24
|
+
server.setRequestHandler(ReadResourceRequestSchema, async () => ({
|
|
25
|
+
contents: []
|
|
26
|
+
}));
|
|
17
27
|
server.setRequestHandler(ListToolsRequestSchema, async () => ({
|
|
18
28
|
tools: toolDefinitions
|
|
19
29
|
}));
|
package/dist/server.js
CHANGED
|
@@ -3,9 +3,11 @@ import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'
|
|
|
3
3
|
import { createServer } from './server-core.js';
|
|
4
4
|
import { getSystemStatus } from './system/index.js';
|
|
5
5
|
const server = createServer();
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
6
|
+
if (process.env.MOBILE_DEBUG_MCP_STARTUP_HEALTHCHECK === '1') {
|
|
7
|
+
getSystemStatus().then((res) => {
|
|
8
|
+
console.info('[startup] system status summary:', { adb: res.adbAvailable, ios: res.iosAvailable, devices: res.devices, iosDevices: res.iosDevices });
|
|
9
|
+
}).catch((e) => console.warn('[startup] healthcheck failed:', e instanceof Error ? e.message : String(e)));
|
|
10
|
+
}
|
|
9
11
|
const transport = new StdioServerTransport();
|
|
10
12
|
async function main() {
|
|
11
13
|
await server.connect(transport);
|
package/docs/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to the **Mobile Debug MCP** project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.30.0]
|
|
6
|
+
- Folded RFC 013 synchronization semantics into the main spec and aligned the interact docs with the shipped `wait_for_ui_change` behavior.
|
|
7
|
+
- Updated `wait_for_ui_change` to use a 300ms stabilization default and to reset stabilization on new in-place mutations.
|
|
8
|
+
- Validated the in-place UI mutation flow on the Modul8 emulator app, including a delayed state-change case.
|
|
9
|
+
|
|
10
|
+
## [0.29.0]
|
|
11
|
+
- Added empty resource handlers and declared the `resources` capability so Codex MCP discovery can complete the handshake against the published npm package.
|
|
12
|
+
- Moved the startup healthcheck behind an opt-in flag to keep the stdio protocol channel quiet by default.
|
|
13
|
+
|
|
5
14
|
## [0.28.0]
|
|
6
15
|
- Added structured execution trace model for all actions within the MCP runtime. It provides visibility into resolution, execution, verification, stabilization, and recovery stages.
|
|
7
16
|
|
package/docs/ROADMAP.md
CHANGED
|
@@ -32,6 +32,7 @@ Track roadmap impact across releases using:
|
|
|
32
32
|
- Gesture success rate
|
|
33
33
|
- Mean time to root cause during debugging
|
|
34
34
|
- Overall agent task completion rate
|
|
35
|
+
- Reduced sequencing errors in multi-step interaction flows
|
|
35
36
|
|
|
36
37
|
Primary KPI:
|
|
37
38
|
Higher task success with fewer retries.
|
|
@@ -45,25 +46,22 @@ Higher task success with fewer retries.
|
|
|
45
46
|
- Stronger State Verification — Complete (Foundational verification layer shipped)
|
|
46
47
|
- Richer Element Identity — Complete (Identity and selector confidence foundations shipped)
|
|
47
48
|
- Better Compose / Custom Control Semantics — Complete (Semantic role enrichment and custom-adjustable inference shipped)
|
|
49
|
+
- Verification Stabilization and Temporal Convergence — Complete (Temporal verification and convergence logic shipped)
|
|
50
|
+
- Action Trace and Execution Observability — Complete (Structured execution trace model shipped)
|
|
51
|
+
- Wait and Synchronization Reliability — Complete (RFC 013 folded into the main spec and shipped behavior verified on emulator)
|
|
48
52
|
|
|
49
53
|
## Current Focus
|
|
50
54
|
|
|
51
|
-
- Wait and Synchronization Reliability
|
|
52
55
|
- Actionability Resolution
|
|
53
|
-
-
|
|
56
|
+
- Adjustable Control Precision Hardening
|
|
54
57
|
|
|
55
58
|
## Upcoming Work
|
|
56
59
|
|
|
57
60
|
- Adjustable Control Precision Hardening
|
|
58
61
|
- Environment Auto-Configuration and Toolchain Discovery
|
|
59
62
|
- Adjustable Control Support
|
|
60
|
-
- Verification Stabilization and Temporal Convergence
|
|
61
63
|
- Signal-Oriented Diagnostic Filtering
|
|
62
64
|
- Long Press Gesture
|
|
63
|
-
# Stronger State Verification
|
|
64
|
-
# Richer Element Identity
|
|
65
|
-
# Wait and Synchronization Reliability
|
|
66
|
-
# Environment Auto-Configuration and Toolchain Discovery
|
|
67
65
|
|
|
68
66
|
## Rationale
|
|
69
67
|
Reduce onboarding friction and improve developer experience by minimizing manual setup dependencies.
|
|
@@ -77,13 +75,14 @@ Addresses friction around:
|
|
|
77
75
|
- environment drift across machines
|
|
78
76
|
- setup failures blocking first use
|
|
79
77
|
|
|
80
|
-
## Scope
|
|
81
78
|
- Automatic discovery of adb
|
|
82
79
|
- Automatic discovery of xcrun
|
|
83
80
|
- idb detection and guided bootstrap support
|
|
84
81
|
- Startup toolchain validation
|
|
85
82
|
- Environment health diagnostics / doctor-style checks
|
|
86
83
|
- Minimal-manual-configuration defaults
|
|
84
|
+
- Runtime device/emulator health signals (crash detection, process lifecycle awareness)
|
|
85
|
+
- App stability monitoring during active sessions
|
|
87
86
|
|
|
88
87
|
## Expected Impact
|
|
89
88
|
High.
|
|
@@ -100,6 +99,7 @@ High.
|
|
|
100
99
|
- Lower environment configuration failures
|
|
101
100
|
- Faster time-to-first-successful-session
|
|
102
101
|
- Reduced support/debugging caused by local setup issues
|
|
102
|
+
- Reduced unknown-failure sessions caused by app or emulator instability
|
|
103
103
|
|
|
104
104
|
## Dependencies
|
|
105
105
|
Depends on:
|
|
@@ -115,7 +115,7 @@ Strengthens:
|
|
|
115
115
|
## Later Horizon
|
|
116
116
|
|
|
117
117
|
- Pinch to Zoom
|
|
118
|
-
-
|
|
118
|
+
- Advanced Trace Correlation and Analysis
|
|
119
119
|
|
|
120
120
|
---
|
|
121
121
|
|
|
@@ -154,7 +154,7 @@ Very high.
|
|
|
154
154
|
Blocks or strengthens:
|
|
155
155
|
- Better Compose / Custom Control Semantics
|
|
156
156
|
- Pinch to Zoom
|
|
157
|
-
-
|
|
157
|
+
- Advanced Trace Correlation and Analysis
|
|
158
158
|
|
|
159
159
|
---
|
|
160
160
|
|
|
@@ -215,8 +215,11 @@ Addresses failures where agents:
|
|
|
215
215
|
- wait_for_ui_change (hierarchy diff based waiting)
|
|
216
216
|
- Structured loading state detection
|
|
217
217
|
- Snapshot revision / staleness metadata
|
|
218
|
-
-
|
|
218
|
+
- Incremental / diff-based snapshot delivery (token-efficient)
|
|
219
|
+
- Focused snapshot scoping (subtree / target-based)
|
|
219
220
|
- Compose-aware wait robustness improvements
|
|
221
|
+
- Explicit interaction sequencing guidance (tap → wait → verify pattern)
|
|
222
|
+
- Exploration of optional action-level synchronization ergonomics (e.g. implicit stabilization or wait flags)
|
|
220
223
|
|
|
221
224
|
## Expected Impact
|
|
222
225
|
Very high.
|
|
@@ -234,6 +237,8 @@ Very high.
|
|
|
234
237
|
- Fewer retries caused by premature actions
|
|
235
238
|
- Higher wait success rate for dynamic UI flows
|
|
236
239
|
- Lower fallback usage to network/log checks
|
|
240
|
+
- Reduced need for manual sequencing by agents in stateful flows
|
|
241
|
+
- Reduced average snapshot size (tokens)
|
|
237
242
|
|
|
238
243
|
## Dependencies
|
|
239
244
|
Depends on:
|
|
@@ -242,7 +247,7 @@ Depends on:
|
|
|
242
247
|
|
|
243
248
|
Blocks or strengthens:
|
|
244
249
|
- Better Compose / Custom Control Semantics
|
|
245
|
-
-
|
|
250
|
+
- Advanced Trace Correlation and Analysis
|
|
246
251
|
|
|
247
252
|
---
|
|
248
253
|
|
|
@@ -251,7 +256,7 @@ Blocks or strengthens:
|
|
|
251
256
|
## Rationale
|
|
252
257
|
Real-world feedback exposed false-negative readiness failures caused by transient UI timing, even when target state had actually converged.
|
|
253
258
|
|
|
254
|
-
**Status:**
|
|
259
|
+
**Status:** Completed
|
|
255
260
|
|
|
256
261
|
Addresses friction where agents:
|
|
257
262
|
- fail readiness checks on transient timing races
|
|
@@ -310,6 +315,7 @@ Addresses cases where:
|
|
|
310
315
|
- Executable-target preference rules
|
|
311
316
|
- Actionability confidence metadata
|
|
312
317
|
- Post-action state verification integration
|
|
318
|
+
- Geometry-aware fallback targeting for weak semantic surfaces (e.g. sliders without accessible nodes)
|
|
313
319
|
|
|
314
320
|
## Expected Impact
|
|
315
321
|
High.
|
|
@@ -324,6 +330,7 @@ High.
|
|
|
324
330
|
- Reduced mis-targeted action failures
|
|
325
331
|
- Lower retarget retries
|
|
326
332
|
- Higher first-attempt action success
|
|
333
|
+
- Reduced need for empirical coordinate probing on custom controls
|
|
327
334
|
|
|
328
335
|
## Dependencies
|
|
329
336
|
Depends on:
|
|
@@ -406,6 +413,7 @@ Addresses friction around:
|
|
|
406
413
|
- Drag vs tap adjustment strategy heuristics
|
|
407
414
|
- Improved value snapping convergence
|
|
408
415
|
- Control-specific adjustment fallback policies
|
|
416
|
+
- Controlled search strategies for value convergence (e.g. binary / progressive adjustment)
|
|
409
417
|
|
|
410
418
|
## Expected Impact
|
|
411
419
|
High.
|
|
@@ -468,7 +476,7 @@ Depends on:
|
|
|
468
476
|
- Wait and Synchronization Reliability
|
|
469
477
|
|
|
470
478
|
Strengthens:
|
|
471
|
-
-
|
|
479
|
+
- Advanced Trace Correlation and Analysis
|
|
472
480
|
|
|
473
481
|
---
|
|
474
482
|
|
|
@@ -615,12 +623,14 @@ Depends on:
|
|
|
615
623
|
|
|
616
624
|
---
|
|
617
625
|
|
|
618
|
-
#
|
|
626
|
+
# Advanced Trace Correlation
|
|
619
627
|
|
|
620
628
|
## Rationale
|
|
621
629
|
Very valuable for debugging,
|
|
622
630
|
but less critical than improving control success first.
|
|
623
631
|
|
|
632
|
+
Builds on the foundational Action Trace and Execution Observability capability by linking traces across UI, network, and logs.
|
|
633
|
+
|
|
624
634
|
**Status:** Planned
|
|
625
635
|
|
|
626
636
|
Improves diagnosis more than task completion.
|
|
@@ -676,7 +686,7 @@ Interaction Expansion
|
|
|
676
686
|
- Pinch to Zoom
|
|
677
687
|
|
|
678
688
|
Deep Observability
|
|
679
|
-
-
|
|
689
|
+
- Advanced Trace Correlation and Analysis
|
|
680
690
|
|
|
681
691
|
## Wave 1 (Current Focus)
|
|
682
692
|
- Stronger State Verification
|
|
@@ -695,7 +705,6 @@ Make core loop reliable and reduce onboarding friction.
|
|
|
695
705
|
- Adjustable Control Precision Hardening
|
|
696
706
|
- Better Compose / Custom Control Semantics
|
|
697
707
|
- Signal-Oriented Diagnostic Filtering
|
|
698
|
-
- Verification Stabilization and Temporal Convergence
|
|
699
708
|
|
|
700
709
|
Focus:
|
|
701
710
|
Improve control precision, verification convergence, custom control reliability, and signal observability.
|
|
@@ -712,7 +721,7 @@ Expand interaction capability after core control reliability is improved.
|
|
|
712
721
|
|
|
713
722
|
## Wave 4 (Advanced Gestures + Deep Observability)
|
|
714
723
|
- Pinch to Zoom
|
|
715
|
-
-
|
|
724
|
+
- Advanced Trace Correlation and Analysis
|
|
716
725
|
|
|
717
726
|
Focus:
|
|
718
727
|
Advanced gestures + deep observability.
|
|
@@ -725,16 +734,15 @@ Roadmap Ordering:
|
|
|
725
734
|
1. Stronger State Verification
|
|
726
735
|
2. Richer Element Identity
|
|
727
736
|
3. Wait and Synchronization Reliability
|
|
728
|
-
4.
|
|
729
|
-
5.
|
|
730
|
-
6.
|
|
731
|
-
7.
|
|
732
|
-
8.
|
|
733
|
-
9.
|
|
734
|
-
10.
|
|
735
|
-
11.
|
|
736
|
-
12.
|
|
737
|
-
13. Action Trace Correlation
|
|
737
|
+
4. Actionability Resolution
|
|
738
|
+
5. Adjustable Control Support
|
|
739
|
+
6. Adjustable Control Precision Hardening
|
|
740
|
+
7. Environment Auto-Configuration and Toolchain Discovery
|
|
741
|
+
8. Better Compose / Custom Control Semantics
|
|
742
|
+
9. Signal-Oriented Diagnostic Filtering
|
|
743
|
+
10. Long Press Gesture
|
|
744
|
+
11. Pinch to Zoom
|
|
745
|
+
12. Advanced Trace Correlation and Analysis
|
|
738
746
|
|
|
739
747
|
Rationale:
|
|
740
748
|
- Early roadmap items harden state, targeting, synchronization, environment readiness, and action execution.
|