mobridge-mcp 0.0.1

package/README.md

@@ -0,0 +1,155 @@
+# mobridge-mcp
+
+**MoBridge** is a Model Context Protocol (MCP) server that gives an AI assistant fast, accurate **eyes and hands on a real Android (and iOS) device** — through the accessibility tree, not screen-scraping. The agent observes the screen as a small semantic model, finds a target, acts on it, and verifies the result. Full-screen pixel screenshots are available when they are genuinely needed.
+
+- **Perception-first.** `screen.observe` returns a lean, deterministic view of what is on screen (app, screen title, actionable controls) — typically ~100× smaller than a full accessibility dump.
+- **Targeted, not brute force.** `screen.find` resolves one target to an act-able `NodeRef` seeded by the platform's own query APIs, with a full-tree oracle as the correctness fallback.
+- **Correctness gates.** Every action re-resolves the target on a fresh capture and is generation-checked: a stale reference fails closed (`STALE_NODE_REF`) instead of acting on the wrong node. No silent "closest match".
+- **Screenshots when you want them.** `screen.capture` returns an inline image (fast WebP by default, lossless PNG on request). `FLAG_SECURE` windows are always refused.
+
+---
+
+## Quick start
+
+```bash
+npx -y mobridge-mcp
+```
+
+That launches the interactive installer, which wires MoBridge into the AI agents you choose. Restart the agent afterwards and ask it to connect to your device.
+
+Requires **Node.js ≥ 18**. For a real Android device you also need **adb** (Android Platform Tools) on your `PATH` and USB debugging enabled.
+
+---
+
+## Installation
+
+### 1. Run the installer
+
+```bash
+npx -y mobridge-mcp
+```
+
+You'll get a menu (powered by `@clack/prompts`):
+
+1. **Pick your agents** — a multi-select of every client the installer can configure (pre-checked if it detects one already installed).
+2. **Pick the platform** — Android or iOS.
+3. **Pick the backend** — `ws` (real device over adb, recommended), `mock` (no device, to explore the tools), or `ios`.
+
+It then writes one `mcpServers` entry into each selected agent's config, **backing up the file first** and touching nothing else. Re-running is safe and idempotent.
+
+Non-interactive (CI / scripted):
+
+```bash
+npx -y mobridge-mcp install --yes \
+  --agent claude-code,cursor,codex --platform android --backend ws
+npx -y mobridge-mcp install --dry-run   # show what it would write, change nothing
+npx -y mobridge-mcp remove              # remove the MoBridge entry again
+```
+
+### 2. Supported agents & config locations
+
+| Agent | Config file | Format |
+|---|---|---|
+| Claude Code (CLI) | `~/.claude.json` | JSON |
+| Claude Desktop | macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`<br>Windows: `%APPDATA%\Claude\claude_desktop_config.json`<br>Linux: `~/.config/Claude/claude_desktop_config.json` | JSON |
+| Cursor | `~/.cursor/mcp.json` | JSON |
+| Windsurf | `~/.codeium/windsurf/mcp_config.json` | JSON |
+| Codex CLI | `~/.codex/config.toml` | TOML |
+
+### 3. Manual installation
+
+If you'd rather edit the config yourself, add this server entry.
+
+**JSON agents** (Claude, Cursor, Windsurf) — merge into `mcpServers`:
+
+```json
+{
+  "mcpServers": {
+    "mobridge": {
+      "command": "npx",
+      "args": ["-y", "mobridge-mcp", "serve"],
+      "env": { "MSM_PLATFORM": "android", "MSM_BACKEND": "ws", "MSM_TOOL_PROFILE": "full" }
+    }
+  }
+}
+```
+
+**Codex CLI** — append to `~/.codex/config.toml`:
+
+```toml
+[mcp_servers.mobridge]
+command = "npx"
+args = ["-y", "mobridge-mcp", "serve"]
+
+[mcp_servers.mobridge.env]
+MSM_PLATFORM = "android"
+MSM_BACKEND = "ws"
+MSM_TOOL_PROFILE = "full"
+```
+
+On **Windows**, MCP clients spawn `npx.cmd`; if a client fails to start the server, use `"command": "cmd"` with `"args": ["/c", "npx", "-y", "mobridge-mcp", "serve"]`. Inside JSON on Windows, escape backslashes in any paths (`\\`).
+
+Restart the agent after editing its config.
+
+---
+
+## Connecting an Android device
+
+1. Enable **Developer options → USB debugging** on the phone and plug it in over USB.
+2. Confirm `adb devices` lists it (accept the RSA prompt on the phone).
+3. Ask the agent to call `mobile.device.connect`. On first connect the companion runtime is provisioned automatically (the on-device app + its accessibility service).
+
+Keep the screen awake during long sessions:
+
+```bash
+adb shell svc power stayon usb
+```
+
+Prefer a wired USB connection — wireless adb links can drop and interrupt the session.
+
+---
+
+## What the agent gets
+
+Progressive disclosure — cheap first, expensive only on demand:
+
+| Tool | Purpose |
+|---|---|
+| `mobile.screen.app` | O(1) — which app/screen am I on? |
+| `mobile.screen.observe` | "AI eyes": a lean list of on-screen text + actionable controls |
+| `mobile.screen.find` / `screen.target` | resolve a target to a `NodeRef` (never coordinates) |
+| `mobile.screen.node` / `screen.children` | inspect one node / one level, on demand |
+| `mobile.screen.act` | re-resolve + act + return before/after evidence for verification |
+| `mobile.screen.capture` | inline screenshot (`preset: "ai"` small & fast, `"full"` lossless) |
+| `mobile.screen.snapshot` | the full accessibility tree — the expensive escape hatch |
+
+The AI stays in charge of *observe → understand → decide → act*. MoBridge only makes the **observe** step fast and precise; it never guesses intent or chains actions on its own.
+
+---
+
+## Privacy & network model
+
+- The device runtime talks to this local MCP server over an adb-forwarded loopback socket. Nothing is sent to third parties.
+- Screenshots stay on device unless a tool call explicitly returns one to the agent; secure (`FLAG_SECURE`) windows are never captured.
+- The installer only writes local agent config files (with backups). It installs no system services and needs no `local.properties`.
+
+---
+
+## Troubleshooting
+
+- **Agent doesn't see the tools** — restart the agent after install; confirm the `mcpServers` entry is present and valid JSON/TOML.
+- **`npx` fails on Windows** — use the `cmd /c npx …` form shown above.
+- **`mobile.device.connect` fails** — check `adb devices`, re-accept the USB debugging prompt, and make sure only one device is attached (or pass the serial).
+- **Empty / one-node captures right after reinstalling the runtime** — toggle the phone's accessibility service off and on, then retry.
+
+Run the server directly (what the agent runs) for a smoke test:
+
+```bash
+npx -y mobridge-mcp serve
+```
+
+---
+
+## License
+
+UNLICENSED — internal distribution. See the repository for terms.

package/dist/adb-baseline/act.mjs

@@ -0,0 +1 @@
+import{execFileSync as u}from"node:child_process";function p(r){const t=r?["-s",r]:[];return e=>u("adb",[...t,...e],{maxBuffer:32*1024*1024})}function l(r){const[t,e,n,o]=r;return[Math.round((t+n)/2),Math.round((e+o)/2)]}function i(r,t){if(!t.bounds)throw new Error("node has no bounds; cannot coordinate-tap");const[e,n]=l(t.bounds);return r(["shell","input","tap",String(e),String(n)]),{method:"GESTURE_FALLBACK",tappedAt:[e,n],target:{role:t.role,label:t.label,text:t.text}}}function s(r,t,e){const[n,o]=l(t.bounds);return r(["shell","input","tap",String(n),String(o)]),r(["shell","input","text",e.replace(/ /g,"%s")]),{method:"GESTURE_FALLBACK",target:{role:t.role,label:t.label}}}function c(r,t){const e={BACK:"KEYCODE_BACK",HOME:"KEYCODE_HOME",RECENTS:"KEYCODE_APP_SWITCH"}[t];if(!e)throw new Error("unsupported global action "+t);return r(["shell","input","keyevent",e]),{method:"GLOBAL_KEY",key:e}}export{l as center,c as globalAction,p as makeAdb,s as setText,i as tapNode};

package/dist/adb-baseline/capture-lib.mjs

@@ -0,0 +1,2 @@
+import{execFileSync as E}from"node:child_process";import{createHash as N}from"node:crypto";function x(t){const n=t?["-s",t]:[];return e=>E("adb",[...n,...e],{maxBuffer:64*1024*1024})}function L(t){const n=/\[(-?\d+),(-?\d+)\]\[(-?\d+),(-?\d+)\]/.exec(t||"");return n?[+n[1],+n[2],+n[3],+n[4]]:null}function R(t){const n={},e=/([\w-]+)="([^"]*)"/g;let i;for(;i=e.exec(t);)n[i[1]]=i[2].replace(/&amp;/g,"&").replace(/&lt;/g,"<").replace(/&gt;/g,">").replace(/&quot;/g,'"');return n}const o=t=>t==="true";function O(t,n){const e=(t||"").toLowerCase();return e.includes("edittext")?"INPUT":e.includes("switch")?"SWITCH":e.includes("checkbox")?"CHECKBOX":e.includes("radio")?"RADIO":e.includes("button")?"BUTTON":e.includes("webview")?"WEB_VIEW":e.includes("recyclerview")||e.includes("listview")||e.includes("scrollview")||e.includes("gridview")?"LIST":e.includes("seekbar")?"SLIDER":e.includes("progress")?"PROGRESS":e.includes("tab")?"TAB":e.includes("imageview")||e.includes("imagebutton")?o(n.clickable)?"BUTTON":"IMAGE":e.includes("textview")?"TEXT":e.includes("edit")?"INPUT":o(n.clickable)?"BUTTON":"UNKNOWN"}function A(t,n){const e=[];return o(n.clickable)&&e.push("CLICK"),o(n["long-clickable"])&&e.push("LONG_CLICK"),t==="INPUT"&&e.push("FOCUS","SET_TEXT","CLEAR_TEXT"),o(n.scrollable)&&e.push("SCROLL_FORWARD","SCROLL_BACKWARD"),o(n.checkable)&&e.push("SELECT"),[...new Set(e)]}function D(t){if(t.role==="SCREEN")return!0;if(t.bounds&&t.bounds[0]===t.bounds[2]&&t.bounds[1]===t.bounds[3])return!1;const e=t.actions.length>0,i=t.text&&t.text.trim()||t.label&&t.label.trim(),r=["LIST","DIALOG","WEB_VIEW","CANVAS"].includes(t.role);return e||i||r}function S({serial:t=null,compact:n=!1}={}){const e=x(t);e(["shell","uiautomator","dump","/sdcard/msm_dump.xml"]);const i=e(["exec-out","cat","/sdcard/msm_dump.xml"]).toString("utf8"),r=[],a=[];let p=0,d=null;const h=/<(\/?)node\b([^>]*?)(\/?)>/g;let f;for(;f=h.exec(i);){if(f[1]==="/"){a.pop();continue}const u=f[3]==="/",s=R(f[2]),b="n"+p++,l=a.length?a[a.length-1]:null;!d&&s.package&&(d=s.package);const c=p===1?"SCREEN":O(s.class,s),g=o(s.password);r.push({nodeId:b,parentId:l,role:c,label:s["content-desc"]||null,text:s.text?s.text:null,value:c==="INPUT"?g?null:s.text||"":null,hint:null,resourceId:s["resource-id"]||null,className:s.class||null,state:{enabled:o(s.enabled),clickable:o(s.clickable),focused:o(s.focused),selected:o(s.selected),checked:s.checkable==="true"?o(s.checked):null,password:g},actions:A(c,s),bounds:L(s.bounds),source:"ACCESSIBILITY",confidence:1}),u||a.push(b)}let w=r;if(n){const u=new Set(r.filter(D).map(l=>l.nodeId)),s=new Map(r.map(l=>[l.nodeId,l])),b=l=>{let c=l.parentId?s.get(l.parentId):null;for(;c&&!u.has(c.nodeId);)c=c.parentId?s.get(c.parentId):null;return c?c.nodeId:null};w=r.filter(l=>u.has(l.nodeId)).map(l=>({...l,parentId:b(l)}))}const T=x(t)(["shell","wm","size"]).toString(),I=/(\d+)x(\d+)/.exec(T),C=I&&+I[1]>+I[2]?"landscape":"portrait",k=w.map(u=>`${u.role}|${u.text}|${u.label}|${u.resourceId}|${(u.bounds||[]).join(",")}`).join(`
+`),m=N("sha1").update(k).digest("hex").slice(0,8);return{schemaVersion:1,deviceId:t||"adb-default",snapshotId:"s_"+Date.now().toString(36),timestamp:Date.now(),packageName:d||"unknown.pkg",windowId:0,orientation:C,screenStateId:(d||"?")+":"+m,compact:n,nodes:w,treeHash:m,_rawNodeCount:r.length}}async function v({serial:t=null,compact:n=!0,timeoutMs:e=6e3,stableForMs:i=600}={}){const r=Date.now();let a=null,p=Date.now();for(;Date.now()-r<e;){const d=S({serial:t,compact:n});if(d.treeHash!==a)a=d.treeHash,p=Date.now();else if(Date.now()-p>=i)return d;await new Promise(h=>setTimeout(h,200))}return S({serial:t,compact:n})}export{S as captureSnapshot,x as makeAdb,v as waitStable};