@mobilenext/mobile-mcp 0.0.10 → 0.0.12

package/README.md

@@ -3,6 +3,9 @@
 This is a [Model Context Protocol (MCP) server](https://github.com/modelcontextprotocol) that enables scalable mobile automation through a platform-agnostic interface, eliminating the need for distinct iOS or Android knowledge.
 This server allows Agents and LLMs to interact with native iOS/Android applications and devices through structured accessibility snapshots or coordinate-based taps based on screenshots. 
 
+https://github.com/user-attachments/assets/c4e89c4f-cc71-4424-8184-bdbc8c638fa1
+
+
 <p align="center">
   <a href="https://www.npmjs.com/package/@mobilenext/mobile-mcp">
     <img src="https://img.shields.io/badge/npm-@mobilenext/mobile--mcp-red" alt="npm">
@@ -18,6 +21,7 @@ This server allows Agents and LLMs to interact with native iOS/Android applicati
     </a>
 </p>
 
+
 ### 🚀 Mobile MCP Roadmap: Building the Future of Mobile
 
 Join us on our journey as we continuously enhance Mobile MCP! 
@@ -46,14 +50,16 @@ How we help to scale mobile automation:
 ## Mobile MCP Architecture
 
 <p align="center">
-    <a href="https://raw.githubusercontent.com/mobile-next/mobile-next-assets/refs/heads/main/mobile-mcp-arch.png">
-        <img alt="mobile-mcp" src="https://raw.githubusercontent.com/mobile-next/mobile-next-assets/refs/heads/main/mobile-mcp-arch.png" width="600">
+    <a href="https://raw.githubusercontent.com/mobile-next/mobile-next-assets/refs/heads/main/mobile-mcp-arch-1.png">
+        <img alt="mobile-mcp" src="https://raw.githubusercontent.com/mobile-next/mobile-next-assets/refs/heads/main/mobile-mcp-arch-1.png" width="600">
     </a>
 </p>
 
 
 
-## How to install
+## Installation and configuration
+
+[Detailed guide for Claude Desktop](https://modelcontextprotocol.io/quickstart/user)
 
 ```json
 {
@@ -67,11 +73,14 @@ How we help to scale mobile automation:
 
 ```
 
-[From Claude Code:](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview)
+[Claude Code:](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview)
+
 ```
 claude mcp add mobile -- npx -y @mobilenext/mobile-mcp@latest ⁠
 ```
 
+[Read more in our wiki](https://github.com/mobile-next/mobile-mcp/wiki)! 🚀
+
 ## Prerequisites
 
 What you will need to connect MCP with your agent and mobile devices:
@@ -102,15 +111,14 @@ On iOS, you'll need Xcode and to run the Simulator before using Mobile MCP with
 - `xcrun simctl list`
 - `xcrun simctl boot "iPhone 16"`
 
-
 # Mobile Commands and interaction tools
 
 The commands and tools support both accessibility-based locators (preferred) and coordinate-based inputs, giving you flexibility when accessibility/automation IDs are missing for reliable and seemless automation.
 
-## mobile_install_app
-- **Description:** Installs an app onto the device/emulator
+## mobile_list_apps
+- **Description:** List all the installed apps on the device
 - **Parameters:**
-  - `appPath` (string): Path or URL to the app file (e.g., .apk for Android, .ipa/.app for iOS)
+  - `bundleId` (string): The application's unique bundle/package identifier like: com.google.android.keep	 or com.apple.mobilenotes )
 
 ## mobile_launch_app
 - **Description:** Launches the specified app on the device/emulator
@@ -120,7 +128,21 @@ The commands and tools support both accessibility-based locators (preferred) and
 ## mobile_terminate_app
 - **Description:** Terminates a running application
 - **Parameters:**
-  - `bundleId` (string): The application's bundle/package identifier
+  - `packageName` (string): Based on the application's bundle/package identifier calls am force stop or kills the app based on pid.
+ 
+## mobile_get_screen_size
+- **Description:** Get the screen size of the mobile device in pixels
+- **Parameters:** None
+
+## mobile_click_on_screen_at_coordinates
+- **Description:** Taps on specified screen coordinates based on coordinates. 
+- **Parameters:**
+  - `x` (number): X-coordinate
+  - `y` (number): Y-coordinate
+ 
+## mobile_list_elements_on_screen
+- **Description:** List elements on screen and their coordinates, with display text or accessibility label.
+- **Parameters:** None
 
 ## mobile_element_tap
 - **Description:** Taps on a UI element identified by accessibility locator
@@ -133,12 +155,19 @@ The commands and tools support both accessibility-based locators (preferred) and
 - **Parameters:**
   - `x` (number): X-coordinate
   - `y` (number): Y-coordinate
+ 
+## mobile_press_button
+- **Description:** Press a button on device (home, back, volume, enter, power button.)
+- **Parameters:** None
+
+## mobile_open_url
+- **Description:** Open a URL in browser on device
+- **Parameters:**
+  - `url` (string): The URL to be opened (e.g., "https://example.com").
 
-## mobile_element_send_keys
-- **Description:** Types text into a UI element (e.g., TextField)
+## mobile_type_text
+- **Description:** Types text into a focused UI element (e.g., TextField, SearchField)
 - **Parameters:**
-  - `element` (string): Human-readable element description
-  - `ref` (string): Accessibility/automation ID of the element
   - `text` (string): Text to type
   - `submit` (boolean): Whether to press Enter/Return after typing
 
@@ -165,20 +194,18 @@ The commands and tools support both accessibility-based locators (preferred) and
 
 ## mobile_take_screenshot
 - **Description:** Captures a screenshot of the current device screen
-- **Parameters:**
-  - `raw` (boolean): Return a lossless image if true; otherwise, compressed by default
+- **Parameters:** None
 
 ## mobile_get_source
 - **Description:** Fetches the current device UI structure (accessibility snapshot) (xml format)
 - **Parameters:** None
 
-## mobile_wait
-- **Description:** Waits for a specified time
-- **Parameters:**
-  - `time` (number): Time to wait in seconds (capped at 10 seconds)
 
-## mobile_close_session
-- **Description:** Closes the current device session
-- **Parameters:** None
+# Thanks to all contributors ❤️
+
+### We appreciate everyone who has helped improve this project. 
 
+  <a href = "https://github.com/mobile-next/mobile-mcp/graphs/contributors">
+   <img src = "https://contrib.rocks/image?repo=mobile-next/mobile-mcp"/>
+ </a>
 

package/lib/android.js

@@ -32,140 +32,168 @@ var __importStar = (this && this.__importStar) || (function () {
         return result;
     };
 })();
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.listApps = exports.takeScreenshot = exports.swipe = exports.getElementsOnScreen = exports.getScreenSize = exports.resolveLaunchableActivities = exports.getConnectedDevices = void 0;
+exports.getConnectedDevices = exports.AndroidRobot = void 0;
+const path_1 = __importDefault(require("path"));
 const child_process_1 = require("child_process");
 const xml = __importStar(require("fast-xml-parser"));
-const fs_1 = require("fs");
-const getConnectedDevices = () => {
-    return (0, child_process_1.execSync)(`adb devices`)
-        .toString()
-        .split("\n")
-        .filter(line => !line.startsWith("List of devices attached"))
-        .filter(line => line.trim() !== "");
+const robot_1 = require("./robot");
+const getAdbPath = () => {
+    let executable = "adb";
+    if (process.env.ANDROID_HOME) {
+        executable = path_1.default.join(process.env.ANDROID_HOME, "platform-tools", "adb");
+    }
+    return executable;
 };
-exports.getConnectedDevices = getConnectedDevices;
-const resolveLaunchableActivities = (packageName) => {
-    return (0, child_process_1.execSync)(`adb shell cmd package resolve-activity ${packageName}`)
-        .toString()
-        .split("\n")
-        .map(line => line.trim())
-        .filter(line => line.startsWith("name="))
-        .map(line => line.substring("name=".length));
+const BUTTON_MAP = {
+    "BACK": "KEYCODE_BACK",
+    "HOME": "KEYCODE_HOME",
+    "VOLUME_UP": "KEYCODE_VOLUME_UP",
+    "VOLUME_DOWN": "KEYCODE_VOLUME_DOWN",
+    "ENTER": "KEYCODE_ENTER",
 };
-exports.resolveLaunchableActivities = resolveLaunchableActivities;
-const getScreenSize = () => {
-    const screenSize = (0, child_process_1.execSync)("adb shell wm size")
-        .toString()
-        .split(" ")
-        .pop();
-    if (!screenSize) {
-        throw new Error("Failed to get screen size");
+const TIMEOUT = 30000;
+const MAX_BUFFER_SIZE = 1024 * 1024 * 4;
+class AndroidRobot {
+    deviceId;
+    constructor(deviceId) {
+        this.deviceId = deviceId;
     }
-    const [width, height] = screenSize.split("x").map(Number);
-    return [width, height];
-};
-exports.getScreenSize = getScreenSize;
-const collectElements = (node, screenSize) => {
-    const elements = [];
-    const getCoordinates = (element) => {
-        const bounds = String(element.bounds);
-        const [, left, top, right, bottom] = bounds.match(/^\[(\d+),(\d+)\]\[(\d+),(\d+)\]$/)?.map(Number) || [];
-        return { left, top, right, bottom };
-    };
-    const getCenter = (coordinates) => {
-        return {
-            x: Math.floor((coordinates.left + coordinates.right) / 2),
-            y: Math.floor((coordinates.top + coordinates.bottom) / 2),
-        };
-    };
-    const normalizeCoordinates = (coordinates, screenSize) => {
-        return {
-            x: Number((coordinates.x / screenSize[0]).toFixed(3)),
-            y: Number((coordinates.y / screenSize[1]).toFixed(3)),
+    adb(...args) {
+        return (0, child_process_1.execFileSync)(getAdbPath(), ["-s", this.deviceId, ...args], {
+            maxBuffer: MAX_BUFFER_SIZE,
+            timeout: TIMEOUT,
+        });
+    }
+    async getScreenSize() {
+        const screenSize = this.adb("shell", "wm", "size")
+            .toString()
+            .split(" ")
+            .pop();
+        if (!screenSize) {
+            throw new Error("Failed to get screen size");
+        }
+        const scale = 1;
+        const [width, height] = screenSize.split("x").map(Number);
+        return { width, height, scale };
+    }
+    async listApps() {
+        return this.adb("shell", "cmd", "package", "query-activities", "-a", "android.intent.action.MAIN", "-c", "android.intent.category.LAUNCHER")
+            .toString()
+            .split("\n")
+            .map(line => line.trim())
+            .filter(line => line.startsWith("packageName="))
+            .map(line => line.substring("packageName=".length))
+            .filter((value, index, self) => self.indexOf(value) === index)
+            .map(packageName => ({
+            packageName,
+            appName: packageName,
+        }));
+    }
+    async launchApp(packageName) {
+        this.adb("shell", "monkey", "-p", packageName, "-c", "android.intent.category.LAUNCHER", "1");
+    }
+    async swipe(direction) {
+        const screenSize = await this.getScreenSize();
+        const centerX = screenSize.width >> 1;
+        // const centerY = screenSize[1] >> 1;
+        let x0, y0, x1, y1;
+        switch (direction) {
+            case "up":
+                x0 = x1 = centerX;
+                y0 = Math.floor(screenSize.height * 0.80);
+                y1 = Math.floor(screenSize.height * 0.20);
+                break;
+            case "down":
+                x0 = x1 = centerX;
+                y0 = Math.floor(screenSize.height * 0.20);
+                y1 = Math.floor(screenSize.height * 0.80);
+                break;
+            default:
+                throw new robot_1.ActionableError(`Swipe direction "${direction}" is not supported`);
+        }
+        this.adb("shell", "input", "swipe", `${x0}`, `${y0}`, `${x1}`, `${y1}`, "1000");
+    }
+    async getScreenshot() {
+        return this.adb("shell", "screencap", "-p");
+    }
+    collectElements(node) {
+        const elements = [];
+        const getScreenElementRect = (element) => {
+            const bounds = String(element.bounds);
+            const [, left, top, right, bottom] = bounds.match(/^\[(\d+),(\d+)\]\[(\d+),(\d+)\]$/)?.map(Number) || [];
+            return {
+                x: left,
+                y: top,
+                width: right - left,
+                height: bottom - top,
+            };
         };
-    };
-    if (node.node) {
-        if (Array.isArray(node.node)) {
-            for (const childNode of node.node) {
-                elements.push(...collectElements(childNode, screenSize));
+        if (node.node) {
+            if (Array.isArray(node.node)) {
+                for (const childNode of node.node) {
+                    elements.push(...this.collectElements(childNode));
+                }
+            }
+            else {
+                elements.push(...this.collectElements(node.node));
             }
         }
-        else {
-            elements.push(...collectElements(node.node, screenSize));
+        if (node.text || node["content-desc"] || node.hint) {
+            const element = {
+                type: node.class || "text",
+                name: node.text,
+                label: node["content-desc"] || node.hint || "",
+                rect: getScreenElementRect(node),
+            };
+            if (element.rect.width > 0 && element.rect.height > 0) {
+                elements.push(element);
+            }
         }
+        return elements;
     }
-    if (node.text) {
-        elements.push({
-            "text": node.text,
-            "coordinates": normalizeCoordinates(getCenter(getCoordinates(node)), screenSize),
+    async getElementsOnScreen() {
+        const dump = this.adb("exec-out", "uiautomator", "dump", "/dev/tty");
+        const parser = new xml.XMLParser({
+            ignoreAttributes: false,
+            attributeNamePrefix: ""
         });
+        const parsedXml = parser.parse(dump);
+        const hierarchy = parsedXml.hierarchy;
+        const elements = this.collectElements(hierarchy.node);
+        return elements;
     }
-    if (node["content-desc"]) {
-        elements.push({
-            "text": node["content-desc"],
-            "coordinates": normalizeCoordinates(getCenter(getCoordinates(node)), screenSize),
-        });
+    async terminateApp(packageName) {
+        this.adb("shell", "am", "force-stop", packageName);
     }
-    return elements;
-};
-const getElementsOnScreen = () => {
-    const dump = (0, child_process_1.execSync)(`adb exec-out uiautomator dump /dev/tty`);
-    const parser = new xml.XMLParser({
-        ignoreAttributes: false,
-        attributeNamePrefix: ""
-    });
-    const parsedXml = parser.parse(dump);
-    const hierarchy = parsedXml.hierarchy;
-    const screenSize = (0, exports.getScreenSize)();
-    const elements = collectElements(hierarchy, screenSize);
-    return elements;
-};
-exports.getElementsOnScreen = getElementsOnScreen;
-const swipe = (direction) => {
-    const screenSize = (0, exports.getScreenSize)();
-    const centerX = screenSize[0] >> 1;
-    // const centerY = screenSize[1] >> 1;
-    let x0, y0, x1, y1;
-    switch (direction) {
-        case "up":
-            x0 = x1 = centerX;
-            y0 = Math.floor(screenSize[1] * 0.80);
-            y1 = Math.floor(screenSize[1] * 0.20);
-            break;
-        case "down":
-            x0 = x1 = centerX;
-            y0 = Math.floor(screenSize[1] * 0.20);
-            y1 = Math.floor(screenSize[1] * 0.80);
-            break;
-        default:
-            throw new Error(`Swipe direction "${direction}" is not supported`);
-    }
-    (0, child_process_1.execSync)(`adb shell input swipe ${x0} ${y0} ${x1} ${y1} 1000`);
-};
-exports.swipe = swipe;
-const takeScreenshot = async () => {
-    const randomFilename = `screenshot-${Date.now()}.png`;
-    // take screenshot and save on device
-    const remoteFilename = `/sdcard/Download/${randomFilename}`;
-    (0, child_process_1.execSync)(`adb shell screencap -p ${remoteFilename}`);
-    // pull the file locally
-    const localFilename = `/tmp/${randomFilename}`;
-    (0, child_process_1.execSync)(`adb pull ${remoteFilename} ${localFilename}`);
-    (0, child_process_1.execSync)(`adb shell rm ${remoteFilename}`);
-    const screenshot = (0, fs_1.readFileSync)(localFilename);
-    (0, fs_1.unlinkSync)(localFilename);
-    return screenshot;
-};
-exports.takeScreenshot = takeScreenshot;
-const listApps = () => {
-    const result = (0, child_process_1.execSync)(`adb shell cmd package query-activities -a android.intent.action.MAIN -c android.intent.category.LAUNCHER`)
+    async openUrl(url) {
+        this.adb("shell", "am", "start", "-a", "android.intent.action.VIEW", "-d", url);
+    }
+    async sendKeys(text) {
+        // adb shell requires some escaping
+        const _text = text.replace(/ /g, "\\ ");
+        this.adb("shell", "input", "text", _text);
+    }
+    async pressButton(button) {
+        if (!BUTTON_MAP[button]) {
+            throw new robot_1.ActionableError(`Button "${button}" is not supported`);
+        }
+        this.adb("shell", "input", "keyevent", BUTTON_MAP[button]);
+    }
+    async tap(x, y) {
+        this.adb("shell", "input", "tap", `${x}`, `${y}`);
+    }
+}
+exports.AndroidRobot = AndroidRobot;
+const getConnectedDevices = () => {
+    return (0, child_process_1.execFileSync)(getAdbPath(), ["devices"])
         .toString()
         .split("\n")
-        .map(line => line.trim())
-        .filter(line => line.startsWith("packageName="))
-        .map(line => line.substring("packageName=".length))
-        .filter((value, index, self) => self.indexOf(value) === index);
-    return result;
+        .filter(line => !line.startsWith("List of devices attached"))
+        .filter(line => line.trim() !== "")
+        .map(line => line.split("\t")[0]);
 };
-exports.listApps = listApps;
+exports.getConnectedDevices = getConnectedDevices;

package/lib/ios.js

@@ -0,0 +1,158 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.IosManager = exports.IosRobot = void 0;
+const path_1 = __importDefault(require("path"));
+const os_1 = require("os");
+const crypto_1 = require("crypto");
+const fs_1 = require("fs");
+const child_process_1 = require("child_process");
+const net_1 = require("net");
+const webdriver_agent_1 = require("./webdriver-agent");
+const robot_1 = require("./robot");
+const WDA_PORT = 8100;
+const IOS_TUNNEL_PORT = 60105;
+const getGoIosPath = () => {
+    if (process.env.GO_IOS_PATH) {
+        return process.env.GO_IOS_PATH;
+    }
+    // fallback to go-ios in PATH via `npm install -g go-ios`
+    return "ios";
+};
+class IosRobot {
+    deviceId;
+    constructor(deviceId) {
+        this.deviceId = deviceId;
+    }
+    isListeningOnPort(port) {
+        return new Promise((resolve, reject) => {
+            const client = new net_1.Socket();
+            client.connect(port, "localhost", () => {
+                client.destroy();
+                resolve(true);
+            });
+            client.on("error", (err) => {
+                resolve(false);
+            });
+        });
+    }
+    async isTunnelRunning() {
+        return await this.isListeningOnPort(IOS_TUNNEL_PORT);
+    }
+    async isWdaForwardRunning() {
+        return await this.isListeningOnPort(WDA_PORT);
+    }
+    async assertTunnelRunning() {
+        if (await this.isTunnelRequired()) {
+            if (!(await this.isTunnelRunning())) {
+                throw new robot_1.ActionableError("iOS tunnel is not running, please see https://github.com/mobile-next/mobile-mcp/wiki/");
+            }
+        }
+    }
+    async wda() {
+        await this.assertTunnelRunning();
+        if (!(await this.isWdaForwardRunning())) {
+            throw new robot_1.ActionableError("Port forwarding to WebDriverAgent is not running (tunnel okay), please see https://github.com/mobile-next/mobile-mcp/wiki/");
+        }
+        const wda = new webdriver_agent_1.WebDriverAgent("localhost", WDA_PORT);
+        if (!(await wda.isRunning())) {
+            throw new robot_1.ActionableError("WebDriverAgent is not running on device (tunnel okay, port forwarding okay), please see https://github.com/mobile-next/mobile-mcp/wiki/");
+        }
+        return wda;
+    }
+    async ios(...args) {
+        return (0, child_process_1.execFileSync)(getGoIosPath(), ["--udid", this.deviceId, ...args], {}).toString();
+    }
+    async getIosVersion() {
+        const output = await this.ios("info");
+        const json = JSON.parse(output);
+        return json.ProductVersion;
+    }
+    async isTunnelRequired() {
+        const version = await this.getIosVersion();
+        const args = version.split(".");
+        return parseInt(args[0], 10) >= 17;
+    }
+    async getScreenSize() {
+        const wda = await this.wda();
+        return await wda.getScreenSize();
+    }
+    async swipe(direction) {
+        const wda = await this.wda();
+        await wda.swipe(direction);
+    }
+    async listApps() {
+        await this.assertTunnelRunning();
+        const output = await this.ios("apps", "--all", "--list");
+        return output
+            .split("\n")
+            .map(line => {
+            const [packageName, appName] = line.split(" ");
+            return {
+                packageName,
+                appName,
+            };
+        });
+    }
+    async launchApp(packageName) {
+        await this.assertTunnelRunning();
+        await this.ios("launch", packageName);
+    }
+    async terminateApp(packageName) {
+        await this.assertTunnelRunning();
+        await this.ios("kill", packageName);
+    }
+    async openUrl(url) {
+        const wda = await this.wda();
+        await wda.openUrl(url);
+    }
+    async sendKeys(text) {
+        const wda = await this.wda();
+        await wda.sendKeys(text);
+    }
+    async pressButton(button) {
+        const wda = await this.wda();
+        await wda.pressButton(button);
+    }
+    async tap(x, y) {
+        const wda = await this.wda();
+        await wda.tap(x, y);
+    }
+    async getElementsOnScreen() {
+        const wda = await this.wda();
+        return await wda.getElementsOnScreen();
+    }
+    async getScreenshot() {
+        await this.assertTunnelRunning();
+        const tmpFilename = path_1.default.join((0, os_1.tmpdir)(), `screenshot-${(0, crypto_1.randomBytes)(8).toString("hex")}.png`);
+        await this.ios("screenshot", "--output", tmpFilename);
+        const buffer = (0, fs_1.readFileSync)(tmpFilename);
+        (0, fs_1.unlinkSync)(tmpFilename);
+        return buffer;
+    }
+}
+exports.IosRobot = IosRobot;
+class IosManager {
+    async isGoIosInstalled() {
+        try {
+            const output = (0, child_process_1.execFileSync)(getGoIosPath(), ["version"], { stdio: ["pipe", "pipe", "ignore"] }).toString();
+            const json = JSON.parse(output);
+            return json.version !== undefined && (json.version.startsWith("v") || json.version === "local-build");
+        }
+        catch (error) {
+            return false;
+        }
+    }
+    async listDevices() {
+        if (!(await this.isGoIosInstalled())) {
+            console.error("go-ios is not installed, no physical iOS devices can be detected");
+            return [];
+        }
+        const output = (0, child_process_1.execFileSync)(getGoIosPath(), ["list"]).toString();
+        const json = JSON.parse(output);
+        return json.deviceList;
+    }
+}
+exports.IosManager = IosManager;

package/lib/iphone-simulator.js

@@ -1,40 +1,127 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.listApps = exports.launchApp = exports.openUrl = exports.getScreenshot = exports.getConnectedDevices = void 0;
+exports.SimctlManager = exports.Simctl = void 0;
 const child_process_1 = require("child_process");
-const getConnectedDevices = () => {
-    return (0, child_process_1.execSync)(`xcrun simctl list devices`)
-        .toString()
-        .split("\n")
-        .map(line => {
-        // extract device name and UUID from the line
-        const match = line.match(/(.*?)\s+\(([\w-]+)\)\s+\(Booted\)/);
-        if (!match) {
-            return null;
+const webdriver_agent_1 = require("./webdriver-agent");
+const robot_1 = require("./robot");
+const TIMEOUT = 30000;
+const WDA_PORT = 8100;
+const MAX_BUFFER_SIZE = 1024 * 1024 * 4;
+class Simctl {
+    simulatorUuid;
+    constructor(simulatorUuid) {
+        this.simulatorUuid = simulatorUuid;
+    }
+    async wda() {
+        const wda = new webdriver_agent_1.WebDriverAgent("localhost", WDA_PORT);
+        if (!(await wda.isRunning())) {
+            throw new robot_1.ActionableError("WebDriverAgent is not running on device (tunnel okay, port forwarding okay), please see https://github.com/mobile-next/mobile-mcp/wiki/");
         }
-        const deviceName = match[1].trim();
-        const deviceUuid = match[2];
-        return {
-            name: deviceName,
-            uuid: deviceUuid,
-        };
-    })
-        .filter(line => line !== null);
-};
-exports.getConnectedDevices = getConnectedDevices;
-const getScreenshot = (simulatorUuid) => {
-    return (0, child_process_1.execSync)(`xcrun simctl io "${simulatorUuid}" screenshot -`);
-};
-exports.getScreenshot = getScreenshot;
-const openUrl = (simulatorUuid, url) => {
-    return (0, child_process_1.execSync)(`xcrun simctl openurl "${simulatorUuid}" "${url}"`);
-};
-exports.openUrl = openUrl;
-const launchApp = (simulatorUuid, packageName) => {
-    return (0, child_process_1.execSync)(`xcrun simctl launch "${simulatorUuid}" "${packageName}"`);
-};
-exports.launchApp = launchApp;
-const listApps = (simulatorUuid) => {
-    return (0, child_process_1.execSync)(`xcrun simctl list apps "${simulatorUuid}"`);
-};
-exports.listApps = listApps;
+        return wda;
+    }
+    simctl(...args) {
+        return (0, child_process_1.execFileSync)("xcrun", ["simctl", ...args], {
+            timeout: TIMEOUT,
+            maxBuffer: MAX_BUFFER_SIZE,
+        });
+    }
+    async getScreenshot() {
+        return this.simctl("io", this.simulatorUuid, "screenshot", "-");
+    }
+    async openUrl(url) {
+        const wda = await this.wda();
+        await wda.openUrl(url);
+        // alternative: this.simctl("openurl", this.simulatorUuid, url);
+    }
+    async launchApp(packageName) {
+        this.simctl("launch", this.simulatorUuid, packageName);
+    }
+    async terminateApp(packageName) {
+        this.simctl("terminate", this.simulatorUuid, packageName);
+    }
+    parseIOSAppData(inputText) {
+        const result = [];
+        // Remove leading and trailing characters if needed
+        const cleanText = inputText.trim();
+        // Extract each app section
+        const appRegex = /"([^"]+)"\s+=\s+\{([^}]+)\};/g;
+        let appMatch;
+        while ((appMatch = appRegex.exec(cleanText)) !== null) {
+            // const bundleId = appMatch[1];
+            const appContent = appMatch[2];
+            const appInfo = {};
+            // parse simple key-value pairs
+            const keyValueRegex = /\s+(\w+)\s+=\s+([^;]+);/g;
+            let keyValueMatch;
+            while ((keyValueMatch = keyValueRegex.exec(appContent)) !== null) {
+                const key = keyValueMatch[1];
+                let value = keyValueMatch[2].trim();
+                // Handle quoted string values
+                if (value.startsWith('"') && value.endsWith('"')) {
+                    value = value.substring(1, value.length - 1);
+                }
+                if (key !== "GroupContainers" && key !== "SBAppTags") {
+                    appInfo[key] = value;
+                }
+            }
+            result.push(appInfo);
+        }
+        return result;
+    }
+    async listApps() {
+        const text = this.simctl("listapps", this.simulatorUuid).toString();
+        const apps = this.parseIOSAppData(text);
+        return apps.map(app => ({
+            packageName: app.CFBundleIdentifier,
+            appName: app.CFBundleDisplayName,
+        }));
+    }
+    async getScreenSize() {
+        const wda = await this.wda();
+        return wda.getScreenSize();
+    }
+    async sendKeys(keys) {
+        const wda = await this.wda();
+        return wda.sendKeys(keys);
+    }
+    async swipe(direction) {
+        const wda = await this.wda();
+        return wda.swipe(direction);
+    }
+    async tap(x, y) {
+        const wda = await this.wda();
+        return wda.tap(x, y);
+    }
+    async pressButton(button) {
+        const wda = await this.wda();
+        return wda.pressButton(button);
+    }
+    async getElementsOnScreen() {
+        const wda = await this.wda();
+        return wda.getElementsOnScreen();
+    }
+}
+exports.Simctl = Simctl;
+class SimctlManager {
+    listSimulators() {
+        const text = (0, child_process_1.execFileSync)("xcrun", ["simctl", "list", "devices", "-j"]).toString();
+        const json = JSON.parse(text);
+        return Object.values(json.devices).flatMap(device => {
+            return device.map(d => {
+                return {
+                    name: d.name,
+                    uuid: d.udid,
+                    state: d.state,
+                };
+            });
+        });
+    }
+    listBootedSimulators() {
+        return this.listSimulators()
+            .filter(simulator => simulator.state === "Booted");
+    }
+    getSimulator(uuid) {
+        return new Simctl(uuid);
+    }
+}
+exports.SimctlManager = SimctlManager;

package/lib/robot.js

@@ -0,0 +1,9 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ActionableError = void 0;
+class ActionableError extends Error {
+    constructor(message) {
+        super(message);
+    }
+}
+exports.ActionableError = ActionableError;

package/lib/server.js

@@ -5,11 +5,13 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.createMcpServer = void 0;
 const mcp_js_1 = require("@modelcontextprotocol/sdk/server/mcp.js");
-const child_process_1 = require("child_process");
-const logger_1 = require("./logger");
 const zod_1 = require("zod");
-const android_1 = require("./android");
 const sharp_1 = __importDefault(require("sharp"));
+const logger_1 = require("./logger");
+const android_1 = require("./android");
+const robot_1 = require("./robot");
+const iphone_simulator_1 = require("./iphone-simulator");
+const ios_1 = require("./ios");
 const getAgentVersion = () => {
     const json = require("../package.json");
     return json.version;
@@ -34,77 +36,136 @@ const createMcpServer = () => {
                 };
             }
             catch (error) {
-                (0, logger_1.trace)(`Tool '${description}' failed: ${error.message} stack: ${error.stack}`);
-                return {
-                    content: [{ type: "text", text: `Error: ${error.message}` }],
-                    isError: true,
-                };
+                if (error instanceof robot_1.ActionableError) {
+                    return {
+                        content: [{ type: "text", text: `${error.message}. Please fix the issue and try again.` }],
+                    };
+                }
+                else {
+                    // a real exception
+                    (0, logger_1.trace)(`Tool '${description}' failed: ${error.message} stack: ${error.stack}`);
+                    return {
+                        content: [{ type: "text", text: `Error: ${error.message}` }],
+                        isError: true,
+                    };
+                }
             }
         };
         server.tool(name, description, paramsSchema, args => wrappedCb(args));
     };
-    tool("list_apps_on_device", "List all apps on device", {}, async ({}) => {
-        const result = (0, android_1.listApps)();
-        return `Found these packages on device: ${result.join(",")}`;
+    let robot;
+    const simulatorManager = new iphone_simulator_1.SimctlManager();
+    const requireRobot = () => {
+        if (!robot) {
+            throw new robot_1.ActionableError("No device selected. Use the mobile_use_device tool to select a device.");
+        }
+    };
+    tool("mobile_list_available_devices", "List all available devices. This includes both physical devices and simulators. If there is more than one device returned, you need to let the user select one of them.", {}, async ({}) => {
+        const iosManager = new ios_1.IosManager();
+        const devices = await simulatorManager.listBootedSimulators();
+        const simulatorNames = devices.map(d => d.name);
+        const androidDevices = (0, android_1.getConnectedDevices)();
+        const iosDevices = await iosManager.listDevices();
+        return `Found these iOS simulators: [${simulatorNames.join(".")}], iOS devices: [${iosDevices.join(",")}] and Android devices: [${androidDevices.join(",")}]`;
+    });
+    tool("mobile_use_device", "Select a device to use. This can be a simulator or an Android device. Use the list_available_devices tool to get a list of available devices.", {
+        device: zod_1.z.string().describe("The name of the device to select"),
+        deviceType: zod_1.z.enum(["simulator", "ios", "android"]).describe("The type of device to select"),
+    }, async ({ device, deviceType }) => {
+        switch (deviceType) {
+            case "simulator":
+                robot = simulatorManager.getSimulator(device);
+                break;
+            case "ios":
+                robot = new ios_1.IosRobot(device);
+                break;
+            case "android":
+                robot = new android_1.AndroidRobot(device);
+                break;
+        }
+        return `Selected device: ${device} (${deviceType})`;
+    });
+    tool("mobile_list_apps", "List all the installed apps on the device", {}, async ({}) => {
+        requireRobot();
+        const result = await robot.listApps();
+        return `Found these apps on device: ${result.map(app => `${app.appName} (${app.packageName})`).join(", ")}`;
     });
-    tool("launch_app", "Launch an app on mobile device", {
+    tool("mobile_launch_app", "Launch an app on mobile device. Use this to open a specific app. You can find the package name of the app by calling list_apps_on_device.", {
         packageName: zod_1.z.string().describe("The package name of the app to launch"),
     }, async ({ packageName }) => {
-        (0, child_process_1.execSync)(`adb shell monkey -p "${packageName}" -c android.intent.category.LAUNCHER 1`);
+        requireRobot();
+        await robot.launchApp(packageName);
         return `Launched app ${packageName}`;
     });
-    tool("terminate_app", "Stop and terminate an app on mobile device", {
+    tool("mobile_terminate_app", "Stop and terminate an app on mobile device", {
         packageName: zod_1.z.string().describe("The package name of the app to terminate"),
     }, async ({ packageName }) => {
-        (0, child_process_1.execSync)(`adb shell am force-stop "${packageName}"`);
+        requireRobot();
+        await robot.terminateApp(packageName);
         return `Terminated app ${packageName}`;
     });
-    tool("get_screen_size", "Get the screen size of the mobile device in pixels", {}, async ({}) => {
-        const screenSize = (0, android_1.getScreenSize)();
-        return `Screen size is ${screenSize[0]}x${screenSize[1]} pixels`;
+    tool("mobile_get_screen_size", "Get the screen size of the mobile device in pixels", {}, async ({}) => {
+        requireRobot();
+        const screenSize = await robot.getScreenSize();
+        return `Screen size is ${screenSize.width}x${screenSize.height} pixels`;
     });
-    tool("click_on_screen_at_coordinates", "Click on the screen at given x,y coordinates", {
-        x: zod_1.z.number().describe("The x coordinate to click between 0 and 1"),
-        y: zod_1.z.number().describe("The y coordinate to click between 0 and 1"),
+    tool("mobile_click_on_screen_at_coordinates", "Click on the screen at given x,y coordinates", {
+        x: zod_1.z.number().describe("The x coordinate to click on the screen, in pixels"),
+        y: zod_1.z.number().describe("The y coordinate to click on the screen, in pixels"),
     }, async ({ x, y }) => {
-        const screenSize = (0, android_1.getScreenSize)();
-        const x0 = Math.floor(screenSize[0] * x);
-        const y0 = Math.floor(screenSize[1] * y);
-        (0, child_process_1.execSync)(`adb shell input tap ${x0} ${y0}`);
+        requireRobot();
+        await robot.tap(x, y);
         return `Clicked on screen at coordinates: ${x}, ${y}`;
     });
-    tool("list_elements_on_screen", "List elements on screen and their coordinates, with display text or accessibility label. Do not cache this result.", {}, async ({}) => {
-        const elements = (0, android_1.getElementsOnScreen)();
-        return `Found these elements on screen: ${JSON.stringify(elements)}`;
+    tool("mobile_list_elements_on_screen", "List elements on screen and their coordinates, with display text or accessibility label. Do not cache this result.", {}, async ({}) => {
+        requireRobot();
+        const elements = await robot.getElementsOnScreen();
+        const result = elements.map(element => {
+            const x = Number((element.rect.x + element.rect.width / 2)).toFixed(3);
+            const y = Number((element.rect.y + element.rect.height / 2)).toFixed(3);
+            return {
+                text: element.label,
+                coordinates: { x, y }
+            };
+        });
+        return `Found these elements on screen: ${JSON.stringify(result)}`;
     });
-    tool("press_button", "Press a button on device", {
-        button: zod_1.z.string().describe("The button to press. Supported buttons: KEYCODE_BACK, KEYCODE_HOME, KEYCODE_MENU, KEYCODE_VOLUME_UP, KEYCODE_VOLUME_DOWN, KEYCODE_ENTER"),
+    tool("mobile_press_button", "Press a button on device", {
+        button: zod_1.z.string().describe("The button to press. Supported buttons: BACK (android only), HOME, VOLUME_UP, VOLUME_DOWN, ENTER"),
     }, async ({ button }) => {
-        (0, child_process_1.execSync)(`adb shell input keyevent ${button}`);
+        requireRobot();
+        await robot.pressButton(button);
         return `Pressed the button: ${button}`;
     });
-    tool("open_url", "Open a URL in browser on device", {
+    tool("mobile_open_url", "Open a URL in browser on device", {
         url: zod_1.z.string().describe("The URL to open"),
     }, async ({ url }) => {
-        (0, child_process_1.execSync)(`adb shell am start -a android.intent.action.VIEW -d "${url}"`);
+        requireRobot();
+        await robot.openUrl(url);
         return `Opened URL: ${url}`;
     });
     tool("swipe_on_screen", "Swipe on the screen", {
         direction: zod_1.z.enum(["up", "down"]).describe("The direction to swipe"),
     }, async ({ direction }) => {
-        (0, android_1.swipe)(direction);
+        requireRobot();
+        await robot.swipe(direction);
         return `Swiped ${direction} on screen`;
     });
-    tool("type_text", "Type text into the focused element", {
+    tool("mobile_type_keys", "Type text into the focused element", {
         text: zod_1.z.string().describe("The text to type"),
-    }, async ({ text }) => {
-        const _text = text.replace(/ /g, "\\ ");
-        (0, child_process_1.execSync)(`adb shell input text "${_text}"`);
+        submit: zod_1.z.boolean().describe("Whether to submit the text. If true, the text will be submitted as if the user pressed the enter key."),
+    }, async ({ text, submit }) => {
+        requireRobot();
+        await robot.sendKeys(text);
+        if (submit) {
+            await robot.pressButton("ENTER");
+        }
         return `Typed text: ${text}`;
     });
-    server.tool("take_device_screenshot", "Take a screenshot of the mobile device. Use this to understand what's on screen, if you need to press an element that is available through view hierarchy then you must list elements on screen instead. Do not cache this result.", {}, async ({}) => {
+    server.tool("mobile_take_screenshot", "Take a screenshot of the mobile device. Use this to understand what's on screen, if you need to press an element that is available through view hierarchy then you must list elements on screen instead. Do not cache this result.", {}, async ({}) => {
+        requireRobot();
         try {
-            const screenshot = await (0, android_1.takeScreenshot)();
+            const screenshot = await robot.getScreenshot();
             // Scale down the screenshot by 50%
             const image = (0, sharp_1.default)(screenshot);
             const metadata = await image.metadata();

package/lib/webdriver-agent.js

@@ -0,0 +1,211 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.WebDriverAgent = void 0;
+const robot_1 = require("./robot");
+class WebDriverAgent {
+    host;
+    port;
+    constructor(host, port) {
+        this.host = host;
+        this.port = port;
+    }
+    async isRunning() {
+        const url = `http://${this.host}:${this.port}/status`;
+        try {
+            const response = await fetch(url);
+            return response.status === 200;
+        }
+        catch (error) {
+            console.error(`Failed to connect to WebDriverAgent: ${error}`);
+            return false;
+        }
+    }
+    async createSession() {
+        const url = `http://${this.host}:${this.port}/session`;
+        const response = await fetch(url, {
+            method: "POST",
+            headers: {
+                "Content-Type": "application/json",
+            },
+            body: JSON.stringify({ capabilities: { alwaysMatch: { platformName: "iOS" } } }),
+        });
+        const json = await response.json();
+        return json.value.sessionId;
+    }
+    async deleteSession(sessionId) {
+        const url = `http://${this.host}:${this.port}/session/${sessionId}`;
+        const response = await fetch(url, { method: "DELETE" });
+        return response.json();
+    }
+    async withinSession(fn) {
+        const sessionId = await this.createSession();
+        const url = `http://${this.host}:${this.port}/session/${sessionId}`;
+        const result = await fn(url);
+        await this.deleteSession(sessionId);
+        return result;
+    }
+    async getScreenSize() {
+        return this.withinSession(async (sessionUrl) => {
+            const url = `${sessionUrl}/wda/screen`;
+            const response = await fetch(url);
+            const json = await response.json();
+            return {
+                width: json.value.screenSize.width,
+                height: json.value.screenSize.height,
+                scale: json.value.scale || 1,
+            };
+        });
+    }
+    async sendKeys(keys) {
+        await this.withinSession(async (sessionUrl) => {
+            const url = `${sessionUrl}/wda/keys`;
+            await fetch(url, {
+                method: "POST",
+                headers: {
+                    "Content-Type": "application/json",
+                },
+                body: JSON.stringify({ value: [keys] }),
+            });
+        });
+    }
+    async pressButton(button) {
+        const _map = {
+            "HOME": "home",
+            "VOLUME_UP": "volumeup",
+            "VOLUME_DOWN": "volumedown",
+        };
+        if (button === "ENTER") {
+            await this.sendKeys("\n");
+            return;
+        }
+        // Type assertion to check if button is a key of _map
+        if (!(button in _map)) {
+            throw new robot_1.ActionableError(`Button "${button}" is not supported`);
+        }
+        await this.withinSession(async (sessionUrl) => {
+            const url = `${sessionUrl}/wda/pressButton`;
+            const response = await fetch(url, {
+                method: "POST",
+                headers: {
+                    "Content-Type": "application/json",
+                },
+                body: JSON.stringify({
+                    name: button,
+                }),
+            });
+            return response.json();
+        });
+    }
+    async tap(x, y) {
+        await this.withinSession(async (sessionUrl) => {
+            const url = `${sessionUrl}/actions`;
+            await fetch(url, {
+                method: "POST",
+                headers: {
+                    "Content-Type": "application/json",
+                },
+                body: JSON.stringify({
+                    actions: [
+                        {
+                            type: "pointer",
+                            id: "finger1",
+                            parameters: { pointerType: "touch" },
+                            actions: [
+                                { type: "pointerMove", duration: 0, x, y },
+                                { type: "pointerDown", button: 0 },
+                                { type: "pause", duration: 100 },
+                                { type: "pointerUp", button: 0 }
+                            ]
+                        }
+                    ]
+                }),
+            });
+        });
+    }
+    isVisible(rect) {
+        return rect.x >= 0 && rect.y >= 0;
+    }
+    filterSourceElements(source) {
+        const output = [];
+        const acceptedTypes = ["TextField", "Button", "Switch", "Icon", "SearchField"];
+        if (acceptedTypes.includes(source.type)) {
+            if (source.isVisible === "1" && this.isVisible(source.rect)) {
+                if (source.label !== null || source.name !== null) {
+                    output.push({
+                        type: source.type,
+                        label: source.label,
+                        name: source.name,
+                        value: source.value,
+                        rect: {
+                            x: source.rect.x,
+                            y: source.rect.y,
+                            width: source.rect.width,
+                            height: source.rect.height,
+                        },
+                    });
+                }
+            }
+        }
+        if (source.children) {
+            for (const child of source.children) {
+                output.push(...this.filterSourceElements(child));
+            }
+        }
+        return output;
+    }
+    async getPageSource() {
+        const url = `http://${this.host}:${this.port}/source/?format=json`;
+        const response = await fetch(url);
+        const json = await response.json();
+        return json;
+    }
+    async getElementsOnScreen() {
+        const source = await this.getPageSource();
+        return this.filterSourceElements(source.value);
+    }
+    async openUrl(url) {
+        await this.withinSession(async (sessionUrl) => {
+            await fetch(`${sessionUrl}/url`, {
+                method: "POST",
+                body: JSON.stringify({ url }),
+            });
+        });
+    }
+    async swipe(direction) {
+        await this.withinSession(async (sessionUrl) => {
+            const x0 = 200;
+            let y0 = 600;
+            const x1 = 200;
+            let y1 = 200;
+            if (direction === "up") {
+                const tmp = y0;
+                y0 = y1;
+                y1 = tmp;
+            }
+            const url = `${sessionUrl}/actions`;
+            await fetch(url, {
+                method: "POST",
+                headers: {
+                    "Content-Type": "application/json",
+                },
+                body: JSON.stringify({
+                    actions: [
+                        {
+                            type: "pointer",
+                            id: "finger1",
+                            parameters: { pointerType: "touch" },
+                            actions: [
+                                { type: "pointerMove", duration: 0, x: x0, y: y0 },
+                                { type: "pointerDown", button: 0 },
+                                { type: "pointerMove", duration: 0, x: x1, y: y1 },
+                                { type: "pause", duration: 1000 },
+                                { type: "pointerUp", button: 0 }
+                            ]
+                        }
+                    ]
+                }),
+            });
+        });
+    }
+}
+exports.WebDriverAgent = WebDriverAgent;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mobilenext/mobile-mcp",
-  "version": "0.0.10",
+  "version": "0.0.12",
   "description": "Mobile MCP",
   "repository": {
     "type": "git",
@@ -13,19 +13,18 @@
   "scripts": {
     "build": "tsc && chmod +x lib/index.js",
     "lint": "eslint .",
+    "test": "nyc mocha --require ts-node/register test/*.ts",
     "watch": "tsc --watch",
-    "clean": "rm -rf lib"
-  },
-  "exports": {
-    "./package.json": "./package.json",
-    ".": {
-      "types": "./index.d.ts",
-      "default": "./index.js"
-    }
+    "clean": "rm -rf lib",
+    "prepare": "husky"
   },
+  "files": [
+    "lib"
+  ],
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.6.1",
     "fast-xml-parser": "^5.0.9",
+    "nyc": "^17.1.0",
     "sharp": "^0.33.5",
     "zod-to-json-schema": "^3.24.4"
   },
@@ -33,6 +32,7 @@
     "@eslint/eslintrc": "^3.2.0",
     "@eslint/js": "^9.19.0",
     "@stylistic/eslint-plugin": "^3.0.1",
+    "@types/mocha": "^10.0.10",
     "@types/node": "^22.13.10",
     "@typescript-eslint/eslint-plugin": "^8.28.0",
     "@typescript-eslint/parser": "^8.26.1",
@@ -41,6 +41,9 @@
     "eslint-plugin": "^1.0.1",
     "eslint-plugin-import": "^2.31.0",
     "eslint-plugin-notice": "^1.0.0",
+    "husky": "^9.1.7",
+    "mocha": "^11.1.0",
+    "ts-node": "^10.9.2",
     "typescript": "^5.8.2"
   },
   "main": "index.js",