mobridge-mcp 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AI-USAGE.md +63 -0
- package/README.md +139 -99
- package/dist/main.obf.cjs +1 -1
- package/dist/protocol/platform-capabilities.json +1 -4
- package/dist/protocol/schemas/action.schema.json +3 -4
- package/dist/protocol/schemas/error.schema.json +2 -1
- package/dist/protocol/schemas/tools.schema.json +7 -2
- package/dist/protocol/schemas/tools.v2.schema.json +55 -0
- package/integrity.json +2 -2
- package/loader.cjs +6 -2
- package/package.json +6 -5
- package/resources/android/app-developer-release.apk +0 -0
- package/resources/ios/MobileSemantic/MobileSemantic.xcodeproj/project.pbxproj +0 -335
- package/resources/ios/MobileSemantic/MobileSemantic.xcodeproj/xcshareddata/xcschemes/MobileSemanticUITests.xcscheme +0 -68
- package/resources/ios/MobileSemantic/UITests/BridgeServer.swift +0 -319
- package/resources/ios/MobileSemantic/UITests/MSHID.h +0 -25
- package/resources/ios/MobileSemantic/UITests/MSHID.m +0 -155
- package/resources/ios/MobileSemantic/UITests/MobileSemanticUITests-Bridging.h +0 -2
- package/resources/ios/MobileSemantic/UITests/XcuiSnapshot.swift +0 -90
- package/resources/ios/MobileSemantic/gen_project.rb +0 -60
package/AI-USAGE.md
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# MoBridge 0.7 — hướng dẫn cho AI
|
|
2
|
+
|
|
3
|
+
MoBridge dùng một daemon dùng chung (`mobd`). CLI và MCP đi qua cùng daemon, vì vậy không mở
|
|
4
|
+
WebSocket riêng và không gọi ADB trực tiếp.
|
|
5
|
+
|
|
6
|
+
## Cài đặt
|
|
7
|
+
|
|
8
|
+
Chạy `npx -y mobridge-mcp install --yes` để tự cấu hình các agent đã phát hiện. Installer backup
|
|
9
|
+
config cũ, gỡ registration MoBridge cũ, cài launcher `~/.mobridge/bin/mobridge`, trích companion
|
|
10
|
+
APK và ghi tài liệu này vào `~/.mobridge/AI-USAGE.md`. Sau đó restart agent.
|
|
11
|
+
|
|
12
|
+
## Chọn CLI hay MCP
|
|
13
|
+
|
|
14
|
+
- Có shell: dùng CLI cho discovery, health/preflight, chẩn đoán, workflow JSONL, output file và
|
|
15
|
+
chuỗi lệnh nhiều bước. Lệnh chính: `mobridge devices`, `connect`, `health`, `observe`, `find`,
|
|
16
|
+
`run`, `trace`.
|
|
17
|
+
- Chỉ có MCP: dùng đúng 5 tool `mobile.device.connect`, `mobile.screen.observe`,
|
|
18
|
+
`mobile.screen.act`, `mobile.screen.capture`, `mobile.run`.
|
|
19
|
+
- MCP dùng cho semantic action và ảnh inline; CLI dùng cho batch/run, stream, diagnostics và file.
|
|
20
|
+
|
|
21
|
+
## Khởi động phiên
|
|
22
|
+
|
|
23
|
+
1. Gọi `mobile.device.connect` hoặc `mobridge connect -d ANDROID:<serial>` một lần.
|
|
24
|
+
2. Luôn đọc `deviceId`, `supportedCapabilities` và trạng thái provision.
|
|
25
|
+
3. Khi có nhiều thiết bị, luôn truyền `deviceId`/`-d`.
|
|
26
|
+
4. Với CLI, preflight là `mobridge health --preflight <package>`.
|
|
27
|
+
5. Mọi response live phải có `backend: "device"`. Nếu thấy `backend: "simulated"`, dừng và không coi kết quả là thao tác trên máy thật. Backend thật là mặc định; fixture chỉ bật tường minh bằng `--simulated` hoặc `MOBRIDGE_BACKEND=sim` cho test.
|
|
28
|
+
|
|
29
|
+
## Quy tắc thao tác nhanh
|
|
30
|
+
|
|
31
|
+
- Kết nối một lần. Nếu đã biết query, gọi thẳng `mobile.screen.act` với `query`; không observe/find riêng.
|
|
32
|
+
- Từ hai bước trở lên dùng `mobile.run`/`mobridge run`; daemon giữ dataflow và session nóng.
|
|
33
|
+
- Chỉ observe khi chưa biết màn hình, sau navigation, khi ambiguous hoặc khi action lỗi.
|
|
34
|
+
- Mỗi MCP act chỉ một verb. Mutation phải có verify; `DISPATCHED` không đồng nghĩa `VERIFIED`.
|
|
35
|
+
- NodeRef phải giữ `snapshotId` và `generation`; NodeRef thuần stale thì dừng, chỉ query gốc mới được resolve lại.
|
|
36
|
+
- `mobile.run` không observe cuối mặc định; dùng `returnState:true` hoặc `mobridge run --return-state` khi cần.
|
|
37
|
+
- Evidence lớn nằm trong trace; response thường chỉ có verdict, state gọn và traceId.
|
|
38
|
+
- Dùng `mobile.run`/`mobridge run` cho dataflow; `$name` chỉ được resolve ở host.
|
|
39
|
+
- Không retry mutation sau transport drop hoặc `RESULT_UNKNOWN`; đọc có thể recovery/retry một lần.
|
|
40
|
+
- Mỗi mutation đơn cần `operationId`; workflow cần `workflowId` và `stepId` ổn định. Khi đổi MCP↔CLI,
|
|
41
|
+
tra operation/workflow status bằng ID gốc trước khi quyết định tiếp. `requestId` chỉ nhận diện một lần gọi.
|
|
42
|
+
- JSONL workflow có checkpoint bền: dùng `mobridge run --status --workflow-id <id>` hoặc `mobile.run`
|
|
43
|
+
với plan `{"control":"status"}`; chỉ resume cùng plan/device bằng token hiện tại.
|
|
44
|
+
- WAL giữ operation tombstone sau 5 phút; unresolved cũ thành `EXPIRED_UNKNOWN`, không trở thành mutation mới.
|
|
45
|
+
Checkpoint workflow tồn tại 24 giờ, chỉ lưu HMAC plan hash và verdict/step IDs, không lưu plan/secret; sau daemon restart hãy status rồi resume bằng token mới.
|
|
46
|
+
- Không gọi `adb`, `uiautomator`, tọa độ hoặc `adb shell input` trong luồng semantic.
|
|
47
|
+
- App control không có authorize/allowlist/denylist theo package. Device identity tường minh, fixed argv,
|
|
48
|
+
safety confirmation và runtime-permission policy vẫn áp dụng; `mobridge app authorize` không tồn tại.
|
|
49
|
+
- Response dùng `transport: "accessibility"` cho semantic/gesture và `transport: "adb"` cho fallback/app step. Gesture tọa độ luôn `verification.level: "unverified"`.
|
|
50
|
+
- Nếu response có `state.interactionBlocked:true` và `state.loadingLikely:true`, UI đang kẹt ở màn hình loading sparse. `mobridge run` sẽ giới hạn wait của bước đó tối đa 2 giây và trả `APP_LOADING_STUCK`; không retry vòng lặp, hãy `capture`/kiểm tra dialog, quyền hoặc mạng rồi observe lại.
|
|
51
|
+
- Secret ưu tiên `valueRef`/secret broker được người dùng cho phép; CLI có thể đọc stdin. Không truyền secret trong argv,
|
|
52
|
+
không dùng environment làm mặc định, không lưu plaintext trong checkpoint, trace hoặc audit.
|
|
53
|
+
- Ad fast path dùng BACK nhưng chỉ xác nhận thành công khi foreground Activity rời ad; nếu chưa rời thì dùng assisted capture/vision.
|
|
54
|
+
|
|
55
|
+
## Ví dụ
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
mobridge devices --json
|
|
59
|
+
mobridge connect -d ANDROID:<serial>
|
|
60
|
+
mobridge tap --text Login --verify auto
|
|
61
|
+
mobridge run login.jsonl -d ANDROID:<serial>
|
|
62
|
+
mobridge trace <traceId>
|
|
63
|
+
```
|
package/README.md
CHANGED
|
@@ -1,155 +1,195 @@
|
|
|
1
1
|
# mobridge-mcp
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Drive a real **Android** phone from an AI agent (or a shell) through the Accessibility tree, not pixels. MoBridge reads the screen as semantic JSON, acts on a target, waits, and **verifies** the effect before it reports success. System work (install, permissions, logs, process control) goes through ADB; pixels are only used when meaning is missing.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
- **Targeted, not brute force.** `screen.find` resolves one target to an act-able `NodeRef` seeded by the platform's own query APIs, with a full-tree oracle as the correctness fallback.
|
|
7
|
-
- **Correctness gates.** Every action re-resolves the target on a fresh capture and is generation-checked: a stale reference fails closed (`STALE_NODE_REF`) instead of acting on the wrong node. No silent "closest match".
|
|
8
|
-
- **Screenshots when you want them.** `screen.capture` returns an inline image (fast WebP by default, lossless PNG on request). `FLAG_SECURE` windows are always refused.
|
|
5
|
+
> Android only. iOS is not supported.
|
|
9
6
|
|
|
10
|
-
|
|
7
|
+
Requires Node.js 18+, Android Platform Tools (`adb`) on `PATH`, and an Android 10+ device (USB debugging on, unlocked for the first connect).
|
|
11
8
|
|
|
12
|
-
##
|
|
9
|
+
## Install
|
|
13
10
|
|
|
14
11
|
```bash
|
|
15
|
-
npx -y mobridge-mcp
|
|
12
|
+
npx -y mobridge-mcp install --yes # wires every detected agent: Claude Code, Claude Desktop, Cursor, Windsurf, Codex
|
|
16
13
|
```
|
|
17
14
|
|
|
18
|
-
|
|
15
|
+
Drop `--yes` for a wizard. The installer backs up each agent config, removes older MoBridge registrations, extracts the signed companion APK, installs a user-owned launcher at `~/.mobridge/bin/mobridge` (add it to `PATH` to use the CLI), writes the agent guide to `~/.mobridge/AI-USAGE.md` and registers `mobridge mcp`. Restart your agent afterwards. `mobridge install --dry-run` previews; `mobridge remove` undoes it.
|
|
19
16
|
|
|
20
|
-
|
|
17
|
+
First connect is automatic: `mobile.device.connect` installs the companion app on the phone, enables its accessibility service through ADB and opens the channel. Nothing else to tap.
|
|
21
18
|
|
|
22
|
-
|
|
19
|
+
## How it works
|
|
23
20
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
```bash
|
|
29
|
-
npx -y mobridge-mcp
|
|
21
|
+
```
|
|
22
|
+
agent ──MCP (5 tools)──┐
|
|
23
|
+
├── mobd daemon ── WebSocket ── companion app (Accessibility)
|
|
24
|
+
shell ──mobridge CLI───┘ └──────── ADB (system actions, fallback)
|
|
30
25
|
```
|
|
31
26
|
|
|
32
|
-
You
|
|
27
|
+
One per-user daemon (`mobd`) serves both the MCP adapter and the CLI, so they share one device lease, one audit log and one workflow state. You never choose a backend: the daemon routes UI work to Accessibility and system work to ADB, and it never switches executor after an ambiguous dispatch.
|
|
33
28
|
|
|
34
|
-
|
|
35
|
-
2. **Pick the platform** — Android or iOS.
|
|
36
|
-
3. **Pick the backend** — `ws` (real device over adb, recommended), `mock` (no device, to explore the tools), or `ios`.
|
|
29
|
+
## The five MCP tools
|
|
37
30
|
|
|
38
|
-
|
|
31
|
+
| Tool | Use it to |
|
|
32
|
+
|---|---|
|
|
33
|
+
| `mobile.device.connect` | Connect one device; returns `deviceId`, capabilities and a ready/blockers/warnings preflight. |
|
|
34
|
+
| `mobile.screen.observe` | Read the screen as compact semantic JSON (no image). |
|
|
35
|
+
| `mobile.screen.act` | Do **one** verified action: click, type, scroll, key, gesture, or click a point on a capture. |
|
|
36
|
+
| `mobile.screen.capture` | Take real pixels (kept in daemon RAM under a `captureId`); also colour search and visual region scan. |
|
|
37
|
+
| `mobile.run` | Run a multi-step JSONL workflow with dataflow, verification, checkpoints and resume. |
|
|
39
38
|
|
|
40
|
-
|
|
39
|
+
### 1. Connect once
|
|
41
40
|
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
--agent claude-code,cursor,codex --platform android --backend ws
|
|
45
|
-
npx -y mobridge-mcp install --dry-run # show what it would write, change nothing
|
|
46
|
-
npx -y mobridge-mcp remove # remove the MoBridge entry again
|
|
41
|
+
```json
|
|
42
|
+
{ "deviceId": "ANDROID:<serial>" }
|
|
47
43
|
```
|
|
48
44
|
|
|
49
|
-
|
|
45
|
+
`deviceId` is optional with one device; with several, pass it on every call or you get `AMBIGUOUS_DEVICE`. Read `deviceId`, `supportedCapabilities` and the provision state from the result. Every live response carries `backend: "device"`; if you ever see `"simulated"`, stop, it is a test fixture and touched no phone.
|
|
46
|
+
|
|
47
|
+
### 2. Observe (only when you do not know the screen)
|
|
50
48
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
| Claude Desktop | macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`<br>Windows: `%APPDATA%\Claude\claude_desktop_config.json`<br>Linux: `~/.config/Claude/claude_desktop_config.json` | JSON |
|
|
55
|
-
| Cursor | `~/.cursor/mcp.json` | JSON |
|
|
56
|
-
| Windsurf | `~/.codeium/windsurf/mcp_config.json` | JSON |
|
|
57
|
-
| Codex CLI | `~/.codex/config.toml` | TOML |
|
|
49
|
+
```json
|
|
50
|
+
{ "actionableOnly": true }
|
|
51
|
+
```
|
|
58
52
|
|
|
59
|
-
|
|
53
|
+
`mode: "fast"` (default) is the cheap survey; `"understand"` adds more context. Observe after navigation, when a result is ambiguous, or after an error. Do **not** observe after every click.
|
|
60
54
|
|
|
61
|
-
|
|
55
|
+
### 3. Act: one verb per call
|
|
62
56
|
|
|
63
|
-
|
|
57
|
+
A target is a **query** (resolved on the device, so no stale reference) or a **NodeRef** copied whole from a previous result. Text fields are objects, not strings.
|
|
58
|
+
|
|
59
|
+
```json
|
|
60
|
+
{ "action": "CLICK", "operationId": "op_login_0001",
|
|
61
|
+
"query": { "text": { "value": "Sign in", "match": "NORMALIZED_EXACT" } },
|
|
62
|
+
"verify": { "policy": "NODE_STATE_CHANGED" } }
|
|
63
|
+
```
|
|
64
64
|
|
|
65
65
|
```json
|
|
66
|
-
{
|
|
67
|
-
"
|
|
68
|
-
"mobridge": {
|
|
69
|
-
"command": "npx",
|
|
70
|
-
"args": ["-y", "mobridge-mcp", "serve"],
|
|
71
|
-
"env": { "MSM_PLATFORM": "android", "MSM_BACKEND": "ws", "MSM_TOOL_PROFILE": "full" }
|
|
72
|
-
}
|
|
73
|
-
}
|
|
74
|
-
}
|
|
66
|
+
{ "action": "SET_TEXT", "operationId": "op_email_0001",
|
|
67
|
+
"query": { "label": { "value": "Email" } }, "text": "a@b.c" }
|
|
75
68
|
```
|
|
76
69
|
|
|
77
|
-
|
|
70
|
+
Other verbs: `SCROLL` (`direction: FORWARD|BACKWARD`), `KEY` (`BACK`, `HOME`, `RECENTS`), `GESTURE` (`TAP|LONG_PRESS|SWIPE|DRAG|PINCH_OPEN|PINCH_CLOSE` with pixel coordinates; last resort, always unverified), and `CLICK` with `at` (below).
|
|
78
71
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
72
|
+
- `operationId` (>= 8 chars) is required for every mutation. It identifies the logical effect: reuse it to look the result up, never to retry.
|
|
73
|
+
- `match` is `EXACT`, `NORMALIZED_EXACT` or `CONTAINS`. Also available in a query: role, resource id, state filters.
|
|
74
|
+
- Many matches returns `AMBIGUOUS_MATCH` with candidates. Narrow the query. MoBridge never clicks the first of several.
|
|
75
|
+
- `verify.policy`: `NODE_STATE_CHANGED`, `NODE_APPEARED`, `NODE_DISAPPEARED`, `NODE_VALUE_EQUALS`, `WINDOW_CHANGED`, `PACKAGE_CHANGED`, or `DISPATCH_ONLY`. Omit it and a sensible one is chosen per verb.
|
|
83
76
|
|
|
84
|
-
|
|
85
|
-
MSM_PLATFORM = "android"
|
|
86
|
-
MSM_BACKEND = "ws"
|
|
87
|
-
MSM_TOOL_PROFILE = "full"
|
|
88
|
-
```
|
|
77
|
+
### 4. Reading a result
|
|
89
78
|
|
|
90
|
-
|
|
79
|
+
| Field / status | Meaning |
|
|
80
|
+
|---|---|
|
|
81
|
+
| `SUCCESS_VERIFIED` | The effect was observed. Trust it. |
|
|
82
|
+
| `VERIFIED_LATE` | The UI rendered after the settle window; a fresh read proved the predicate. Still a success, no replay happened. |
|
|
83
|
+
| `ACTION_DISPATCHED` / `DISPATCHED_NO_VERIFIED_EFFECT` | The input was accepted but its effect is unknown (always the case for coordinate gestures). Observe before continuing. |
|
|
84
|
+
| `RESULT_UNKNOWN` | The link dropped after dispatch. **Do not repeat the action.** Call `mobridge operation status <operationId>` (or `mobile.run` with plan `{"control":"status"}`). |
|
|
85
|
+
| `AMBIGUOUS_MATCH` / `AMBIGUOUS_VISUAL` | More than one candidate. Narrow the query or look at a capture. |
|
|
86
|
+
| `STALE_NODE_REF` / `STALE_CAPTURE` | The screen moved on. Re-observe or re-capture and choose again. |
|
|
87
|
+
| `NEEDS_VISION` | Semantics are not enough. Use `mobile.screen.capture`, look, then act with `at`. |
|
|
88
|
+
| `USER_CONFIRMATION_REQUIRED` | A safety gate wants the user. Stop and ask. |
|
|
91
89
|
|
|
92
|
-
|
|
90
|
+
The agent only needs to come back to the model for: ambiguity, `NEEDS_VISION`, `RESULT_UNKNOWN`, confirmation or unsupported operations. Everything else is deterministic.
|
|
93
91
|
|
|
94
|
-
|
|
92
|
+
### 5. Capture: when meaning is missing
|
|
95
93
|
|
|
96
|
-
|
|
94
|
+
`mobile.screen.capture` returns an image plus a `captureId`. The pixels stay in daemon memory (last 5 captures, about 30 s, never written to disk). Default is a small preview; for fine detail pass `roi` (`x,y,w,h` in screen pixels) at native resolution instead of re-capturing the whole screen.
|
|
97
95
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
96
|
+
- **Click what you see.** Choose `x,y` in capture-image pixels and call act `CLICK` with `at: { captureId, x, y }`. A node under the point is clicked and verified; otherwise one guarded gesture is sent. Use `stalePolicy: "VISUAL_DYNAMIC"` for ads, video and canvas, whose tree changes every frame; the default `STRICT` refuses if the screen changed.
|
|
97
|
+
- **Find by colour.** `color: { hex: "#1A73E8", tolerance: 60 }` returns `colorMatch` (`ONE` with a ref, `AMBIGUOUS`, `NONE`, `NEEDS_VISION`) measured on the device bitmap.
|
|
98
|
+
- **Find controls nothing declares.** `scan: {}` (optional `zone: [l,t,r,b]`) returns `regionScan.regions[]`: compact visual components with `regionId`, `bounds`, `center`, `position` (3x3 cell), `aspectRatio`, `edgeDensity`, `contrast`, `semanticNode` (the labelled/actionable node under it, or `null`) and a ready-to-use `at`. System bars and FLAG_SECURE windows are excluded. MoBridge names no icon and picks nothing: look at the image, choose a region, act on its `at`.
|
|
99
|
+
- Secure windows (`FLAG_SECURE`) block capture only (`CAPTURE_BLOCKED_SECURE_WINDOW`); semantic observe/act keeps working.
|
|
101
100
|
|
|
102
|
-
|
|
101
|
+
### 6. Workflows with `mobile.run`
|
|
103
102
|
|
|
104
|
-
|
|
105
|
-
|
|
103
|
+
Two or more steps: send a JSONL plan instead of many calls. The daemon keeps the dataflow, verifies each mutation and checkpoints progress.
|
|
104
|
+
|
|
105
|
+
```json
|
|
106
|
+
{ "workflowId": "wf_login_0001",
|
|
107
|
+
"plan": "{\"id\":\"u\",\"type\":{\"field\":\"Email\",\"text\":\"a@b.c\"}}\n{\"id\":\"go\",\"tap\":\"Sign in\"}\n" }
|
|
106
108
|
```
|
|
107
109
|
|
|
108
|
-
|
|
110
|
+
- Steps: `tap`, `type`, `scroll`, `key`, `gesture`, `capture` (`{"pause":true}` yields `NEEDS_VISION` with a `resumeToken`), `op: app|permission|package|...`, `wait`, plus `when` / `unless` conditions on `package` and `dialogKind`.
|
|
111
|
+
- `$name` results are resolved on the host; secrets go through `valueRef`/stdin, never argv or plan text.
|
|
112
|
+
- A failed or unverified mutation stops the run (`PAUSED`); nothing is replayed automatically. Resume with the same `workflowId`, plan, device and the `resumeToken`.
|
|
113
|
+
- Status without re-running: plan `{"control":"status"}` (CLI: `mobridge run --status --workflow-id <id>`). Checkpoints live 24 h and hold only hashes and step ids.
|
|
114
|
+
- The final screen is omitted unless `returnState: true`. `waitUntil` replaces any watch/stream.
|
|
109
115
|
|
|
110
|
-
|
|
116
|
+
## Recipes
|
|
111
117
|
|
|
112
|
-
|
|
118
|
+
| Goal | How |
|
|
119
|
+
|---|---|
|
|
120
|
+
| Open an app and tap something | CLI `mobridge open <package>` then `act` with a query. Do not observe in between. |
|
|
121
|
+
| Fill a form | One `mobile.run` plan with `type` steps and a final `tap`. |
|
|
122
|
+
| Answer a permission dialog | CLI `mobridge permission respond --decision ALLOW_WHILE_USING` (answers the dialog like a user). `permission set <pkg> CAMERA` changes the state directly via ADB and reads it back. They are not interchangeable. |
|
|
123
|
+
| Close an ad | CLI `mobridge ad dismiss`. It tries BACK, then assisted capture; it never taps a call-to-action. Exit 8 = pick the close point from the returned capture, then `mobridge ad tap-close`. |
|
|
124
|
+
| Install / force-stop / uninstall | CLI `mobridge package install|uninstall`, `mobridge process stop`; each reads the package/process state back. |
|
|
125
|
+
| Diagnose | CLI `mobridge diag LOGCAT|CRASH|BATTERY|NETWORK|DISPLAY|POWER`. |
|
|
126
|
+
| Anything missing | `mobridge capabilities [--family F]` lists every operation, its executor, replay class and verification. |
|
|
127
|
+
|
|
128
|
+
## Rules that keep it safe
|
|
129
|
+
|
|
130
|
+
1. Connect once, then act directly when you know the target. Observe only to learn the screen.
|
|
131
|
+
2. One mutation, one `operationId`. After `RESULT_UNKNOWN` look the operation up; never resend, and never fall back to raw ADB.
|
|
132
|
+
3. `DISPATCHED` is not `VERIFIED`. Check `verification` before assuming success.
|
|
133
|
+
4. NodeRefs are bound to a screen generation; a stale one fails closed instead of guessing.
|
|
134
|
+
5. Do not call `adb`, `uiautomator` or coordinate taps from the agent; use the typed tools.
|
|
135
|
+
6. Coordinates from a capture are valid only for that capture (`STALE_CAPTURE` otherwise).
|
|
113
136
|
|
|
114
|
-
|
|
137
|
+
## Troubleshooting
|
|
115
138
|
|
|
116
|
-
|
|
|
139
|
+
| Symptom | Fix |
|
|
117
140
|
|---|---|
|
|
118
|
-
| `
|
|
119
|
-
| `
|
|
120
|
-
| `
|
|
121
|
-
| `
|
|
122
|
-
| `
|
|
123
|
-
|
|
|
124
|
-
| `mobile.screen.snapshot` | the full accessibility tree — the expensive escape hatch |
|
|
141
|
+
| `AMBIGUOUS_DEVICE` | Pass `deviceId` (`mobridge devices`). |
|
|
142
|
+
| `COMPONENT_MISSING` / `COMPANION_UNAVAILABLE` | Run `mobridge connect -d ANDROID:<serial>` again; it reinstalls and re-enables the companion. |
|
|
143
|
+
| `ACCESSIBILITY_PREEMPTED` / `EXTERNAL_INSTRUMENTATION_ACTIVE` | Another tool (UiAutomator, a test runner, another agent) owns accessibility. Stop it and reconnect. |
|
|
144
|
+
| `TRANSPORT_DEGRADED` | Link dropped; prefer USB over wireless adb. `mobridge health` shows the socket state. |
|
|
145
|
+
| `UNSUPPORTED_PLATFORM` | You asked for iOS. Only Android is supported. |
|
|
146
|
+
| Companion disables itself | By design after 5 minutes without a controller; the next connect re-arms it. |
|
|
125
147
|
|
|
126
|
-
|
|
148
|
+
## CLI contract
|
|
127
149
|
|
|
128
|
-
|
|
150
|
+
stdout is one JSON line when non-interactive (or with `--json`); logs go to stderr. `-d/--device` is accepted by every device command. With multiple devices, omitting it returns `AMBIGUOUS_DEVICE`.
|
|
129
151
|
|
|
130
|
-
|
|
152
|
+
```text
|
|
153
|
+
0 verified/read OK 1 known failure 2 dispatched/unverified
|
|
154
|
+
3 result unknown/cancelled after dispatch 4 hard gate
|
|
155
|
+
5 not found/ambiguous 6 safety/token block 7 not dispatched/retryable transport
|
|
156
|
+
```
|
|
131
157
|
|
|
132
|
-
|
|
133
|
-
- Screenshots stay on device unless a tool call explicitly returns one to the agent; secure (`FLAG_SECURE`) windows are never captured.
|
|
134
|
-
- The installer only writes local agent config files (with backups). It installs no system services and needs no `local.properties`.
|
|
158
|
+
Examples:
|
|
135
159
|
|
|
136
|
-
|
|
160
|
+
```bash
|
|
161
|
+
mobridge find --role button --text Save | mobridge screen act click - --verify auto
|
|
162
|
+
mobridge screen act type --field email 'a@b.c' --verify auto
|
|
163
|
+
mobridge screen act type --field password --secret-env LOGIN_PASSWORD --verify auto
|
|
164
|
+
mobridge run login.jsonl -d ANDROID:<serial>
|
|
165
|
+
mobridge trace <traceId>
|
|
166
|
+
```
|
|
137
167
|
|
|
138
|
-
|
|
168
|
+
`mobridge run` accepts JSONL dataflow. Results are resolved on the host (`$name`), never sent as literal placeholders to the device. Mutation steps require verification; the first error stops the plan. Reads may recover and retry once; mutations never replay automatically.
|
|
139
169
|
|
|
140
|
-
|
|
141
|
-
- **`npx` fails on Windows** — use the `cmd /c npx …` form shown above.
|
|
142
|
-
- **`mobile.device.connect` fails** — check `adb devices`, re-accept the USB debugging prompt, and make sure only one device is attached (or pass the serial).
|
|
143
|
-
- **Empty / one-node captures right after reinstalling the runtime** — toggle the phone's accessibility service off and on, then retry.
|
|
170
|
+
## Transport and ADB
|
|
144
171
|
|
|
145
|
-
|
|
172
|
+
`mobd` keeps one stable controller lease. The application heartbeat is `device.status` every 60 seconds; WebSocket ping/pong runs at 5/10 seconds. Recovery is close → ADB remote connect when required → verify/create owned forward → re-enable accessibility → handshake → health. Recovery never installs an APK; missing components return `COMPONENT_MISSING` and suggest `mobridge connect`. The old semantic `MSM_BACKEND=adb` path is disabled; bounded ADB gestures are available only through a prepared fallback token.
|
|
146
173
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
174
|
+
MoBridge forwards use `adb -s <serial> forward --no-rebind` on ports 18765–18865. Owned mappings persist in `~/.mobridge/state/ports.json`; unrelated forwards are never removed. Every ADB call has an explicit timeout and commands are serialized per serial.
|
|
175
|
+
|
|
176
|
+
`mobridge health` reports server PID/version, port, WebSocket state, close code, heartbeat, serial and device state. `mobridge doctor --restart-adb` is the only command allowed to restart the ADB server.
|
|
177
|
+
|
|
178
|
+
## Audit and safety
|
|
150
179
|
|
|
151
|
-
|
|
180
|
+
Mutations use write-ahead audit (`intent` fsync before dispatch, `outcome` after). An intent without an outcome becomes `RESULT_UNKNOWN` after daemon restart and is reported as `orphanedMutations`. Host payloads are hashed/redacted; files are mode 0600, rotated daily, retained seven days and capped at 100 MB.
|
|
152
181
|
|
|
153
|
-
|
|
182
|
+
Coordinate/ADB gestures require an explicit fallback token and a post-action observe. `type --secret` reads stdin or an environment variable and never places the secret in argv, audit, trace or error output. The daemon rejects raw `uiautomator` execution.
|
|
183
|
+
|
|
184
|
+
## Development
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
npm run build
|
|
188
|
+
npm run gate:adb-safety
|
|
189
|
+
npm run gate:cli-contract
|
|
190
|
+
npm run test:act-find
|
|
191
|
+
npm run test:mutation-identity
|
|
192
|
+
npm run test:stale-recovery
|
|
193
|
+
```
|
|
154
194
|
|
|
155
|
-
|
|
195
|
+
The packaged smoke gate must run against the actual packaged artifact. Live fault-injection runs record attached devices; if Samsung is not discovered its status is `NOT_RUN`, never `PASS`.
|