@mobilenext/mobile-mcp 0.0.12 → 0.0.14

package/README.md

@@ -1,6 +1,6 @@
-## Mobile Next - MCP server for Mobile Automation
+# Mobile Next - MCP server for Mobile Development and Automation  | iOS, Android, Simulator, Emulator, and physical devices
 
-This is a [Model Context Protocol (MCP) server](https://github.com/modelcontextprotocol) that enables scalable mobile automation through a platform-agnostic interface, eliminating the need for distinct iOS or Android knowledge.
+This is a [Model Context Protocol (MCP) server](https://github.com/modelcontextprotocol) that enables scalable mobile automation, development through a platform-agnostic interface, eliminating the need for distinct iOS or Android knowledge. You can run it on emulators, simulators, and physical devices (iOS and Android).
 This server allows Agents and LLMs to interact with native iOS/Android applications and devices through structured accessibility snapshots or coordinate-based taps based on screenshots. 
 
 https://github.com/user-attachments/assets/c4e89c4f-cc71-4424-8184-bdbc8c638fa1
@@ -27,7 +27,8 @@ https://github.com/user-attachments/assets/c4e89c4f-cc71-4424-8184-bdbc8c638fa1
 Join us on our journey as we continuously enhance Mobile MCP! 
 Check out our detailed roadmap to see upcoming features, improvements, and milestones. Your feedback is invaluable in shaping the future of mobile automation.
 
-👉 [Explore the Roadmap](https://github.com/orgs/mobile-next/projects/1)
+👉 [Explore the Roadmap](https://github.com/orgs/mobile-next/projects/3)
+
 
 ### Main use cases
 
@@ -47,7 +48,7 @@ How we help to scale mobile automation:
 - 📊 **Deterministic tool application**: Reduces ambiguity found in purely screenshot-based approaches by relying on structured data whenever possible.
 - 📺 **Extract structured data**: Enables you to extract structred data from anything visible on screen. 
 
-## Mobile MCP Architecture
+## 🏗️ Mobile MCP Architecture
 
 <p align="center">
     <a href="https://raw.githubusercontent.com/mobile-next/mobile-next-assets/refs/heads/main/mobile-mcp-arch-1.png">
@@ -56,10 +57,14 @@ How we help to scale mobile automation:
 </p>
 
 
+## 📚 Wiki page 
+
+More details in our [wiki page](https://github.com/mobile-next/mobile-mcp/wiki) for setup, configuration and debugging related questions.
+
 
 ## Installation and configuration
 
-[Detailed guide for Claude Desktop](https://modelcontextprotocol.io/quickstart/user)
+Setup our MCP with Cursor, Claude, VS Code, Github Copilot:
 
 ```json
 {
@@ -81,6 +86,64 @@ claude mcp add mobile -- npx -y @mobilenext/mobile-mcp@latest ⁠
 
 [Read more in our wiki](https://github.com/mobile-next/mobile-mcp/wiki)! 🚀
 
+
+### 🛠️ How to Use 📝
+
+After adding the MCP server to your IDE/Client, you can instruct your AI assistant to use the available tools. 
+For example, in Cursor's agent mode, you could use the prompts below to quickly validate, test and iterate on UI intereactions, read information from screen, go through complex workflows.
+Be descriptive, straight to the point.
+
+### ✨ Example Prompts
+
+#### Workflows
+
+You can specifiy detailed workflows in a single prompt, verify business logic, setup automations. You can go crazy:
+
+**Search for a video, comment, like and share it.**
+```
+Find the video called " Beginner Recipe for Tonkotsu Ramen" by Way of Ramen, click on like video, after liking write a comment " this was delicious, will make it next Friday", share the video with the first contact in your whatsapp list. 
+```
+
+**Download a successful step counter app, register, setup workout and 5 start the app**
+```
+Find and Download a free "Pomodoro" app that has more thank 1k stars. 
+Launch the app, register with my email, after registration find how to start a pomodoro timer. 
+When the pomodoro timer started, go back to the app store and rate the app 5 stars, 
+and leave a comment how useful the app is. 
+```
+
+**Search in Substack, read, highlight, comment and save an article**
+```
+Open Substack website, search for "Latest trends in AI automation 2025", open the first article, 
+highlight the section titled "Emerging AI trends", and save article to reading list for later review, 
+comment a random paragraph summary.
+```
+
+**Reserve a workout class, set timer**
+```
+Open ClassPass, search for yoga classes tomorrow morning within 2 miles, 
+book the highest-rated class at 7 AM, confirm reservation,
+ setup a timer for the booked slot in the phone
+```
+
+**Find a local event, setup calendar event**
+```
+Open Eventbrite, search for AI startup meetup events happening this weekend in "Austin, TX", 
+select the most popular one, register and RSVP yes to the even, setup a calendar event as a reminder.
+```
+
+**Check weather forecast and send a Whatsapp/Telegram/Slack message**
+```
+Open Weather app, check tomorrow's weather forecast for "Berlin", and send the summary 
+via Whatsapp/Telegram/Slack to contact "Lauren Trown", thumbs up their response.
+```
+
+- **Schedule a meeting in Zoom and share invite via email**
+```
+Open Zoom app, schedule a meeting titled "AI Hackathon" for tomorrow at 10 AM with a duration of 1 hour,
+copy the invitation link, and send it via Gmail to contacts "team@example.com".
+```
+
 ## Prerequisites
 
 What you will need to connect MCP with your agent and mobile devices:
@@ -111,96 +174,6 @@ On iOS, you'll need Xcode and to run the Simulator before using Mobile MCP with
 - `xcrun simctl list`
 - `xcrun simctl boot "iPhone 16"`
 
-# Mobile Commands and interaction tools
-
-The commands and tools support both accessibility-based locators (preferred) and coordinate-based inputs, giving you flexibility when accessibility/automation IDs are missing for reliable and seemless automation.
-
-## mobile_list_apps
-- **Description:** List all the installed apps on the device
-- **Parameters:**
-  - `bundleId` (string): The application's unique bundle/package identifier like: com.google.android.keep	 or com.apple.mobilenotes )
-
-## mobile_launch_app
-- **Description:** Launches the specified app on the device/emulator
-- **Parameters:**
-  - `bundleId` (string): The application's unique bundle/package identifier like: com.google.android.keep	 or com.apple.mobilenotes )
-
-## mobile_terminate_app
-- **Description:** Terminates a running application
-- **Parameters:**
-  - `packageName` (string): Based on the application's bundle/package identifier calls am force stop or kills the app based on pid.
- 
-## mobile_get_screen_size
-- **Description:** Get the screen size of the mobile device in pixels
-- **Parameters:** None
-
-## mobile_click_on_screen_at_coordinates
-- **Description:** Taps on specified screen coordinates based on coordinates. 
-- **Parameters:**
-  - `x` (number): X-coordinate
-  - `y` (number): Y-coordinate
- 
-## mobile_list_elements_on_screen
-- **Description:** List elements on screen and their coordinates, with display text or accessibility label.
-- **Parameters:** None
-
-## mobile_element_tap
-- **Description:** Taps on a UI element identified by accessibility locator
-- **Parameters:**
-  - `element` (string): Human-readable element description (e.g., "Login button")
-  - `ref` (string): Accessibility/automation ID or reference from a snapshot
-
-## mobile_tap
-- **Description:** Taps on specified screen coordinates
-- **Parameters:**
-  - `x` (number): X-coordinate
-  - `y` (number): Y-coordinate
- 
-## mobile_press_button
-- **Description:** Press a button on device (home, back, volume, enter, power button.)
-- **Parameters:** None
-
-## mobile_open_url
-- **Description:** Open a URL in browser on device
-- **Parameters:**
-  - `url` (string): The URL to be opened (e.g., "https://example.com").
-
-## mobile_type_text
-- **Description:** Types text into a focused UI element (e.g., TextField, SearchField)
-- **Parameters:**
-  - `text` (string): Text to type
-  - `submit` (boolean): Whether to press Enter/Return after typing
-
-## mobile_element_swipe
-- **Description:** Performs a swipe gesture from one UI element to another
-- **Parameters:**
-  - `startElement` (string): Human-readable description of the start element
-  - `startRef` (string): Accessibility/automation ID of the start element
-  - `endElement` (string): Human-readable description of the end element
-  - `endRef` (string): Accessibility/automation ID of the end element
- 
-## mobile_swipe
-- **Description:** Performs a swipe gesture between two sets of screen coordinates
-- **Parameters:**
-  - `startX` (number): Start X-coordinate
-  - `startY` (number): Start Y-coordinate
-  - `endX` (number): End X-coordinate
-  - `endY` (number): End Y-coordinate
-
-## mobile_press_key
-- **Description:** Presses hardware keys or triggers special events (e.g., back button on Android)
-- **Parameters:**
-  - `key` (string): Key identifier (e.g., HOME, BACK, VOLUME_UP, etc.)
-
-## mobile_take_screenshot
-- **Description:** Captures a screenshot of the current device screen
-- **Parameters:** None
-
-## mobile_get_source
-- **Description:** Fetches the current device UI structure (accessibility snapshot) (xml format)
-- **Parameters:** None
-
-
 # Thanks to all contributors ❤️
 
 ### We appreciate everyone who has helped improve this project. 

package/lib/android.js

@@ -36,7 +36,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.getConnectedDevices = exports.AndroidRobot = void 0;
+exports.AndroidDeviceManager = exports.AndroidRobot = void 0;
 const path_1 = __importDefault(require("path"));
 const child_process_1 = require("child_process");
 const xml = __importStar(require("fast-xml-parser"));
@@ -54,6 +54,11 @@ const BUTTON_MAP = {
     "VOLUME_UP": "KEYCODE_VOLUME_UP",
     "VOLUME_DOWN": "KEYCODE_VOLUME_DOWN",
     "ENTER": "KEYCODE_ENTER",
+    "DPAD_CENTER": "KEYCODE_DPAD_CENTER",
+    "DPAD_UP": "KEYCODE_DPAD_UP",
+    "DPAD_DOWN": "KEYCODE_DPAD_DOWN",
+    "DPAD_LEFT": "KEYCODE_DPAD_LEFT",
+    "DPAD_RIGHT": "KEYCODE_DPAD_RIGHT",
 };
 const TIMEOUT = 30000;
 const MAX_BUFFER_SIZE = 1024 * 1024 * 4;
@@ -68,6 +73,14 @@ class AndroidRobot {
             timeout: TIMEOUT,
         });
     }
+    getSystemFeatures() {
+        return this.adb("shell", "pm", "list", "features")
+            .toString()
+            .split("\n")
+            .map(line => line.trim())
+            .filter(line => line.startsWith("feature:"))
+            .map(line => line.substring("feature:".length));
+    }
     async getScreenSize() {
         const screenSize = this.adb("shell", "wm", "size")
             .toString()
@@ -96,6 +109,14 @@ class AndroidRobot {
     async launchApp(packageName) {
         this.adb("shell", "monkey", "-p", packageName, "-c", "android.intent.category.LAUNCHER", "1");
     }
+    async listRunningProcesses() {
+        return this.adb("shell", "ps", "-e")
+            .toString()
+            .split("\n")
+            .map(line => line.trim())
+            .filter(line => line.startsWith("u")) // non-system processes
+            .map(line => line.split(/\s+/)[8]); // get process name
+    }
     async swipe(direction) {
         const screenSize = await this.getScreenSize();
         const centerX = screenSize.width >> 1;
@@ -122,16 +143,6 @@ class AndroidRobot {
     }
     collectElements(node) {
         const elements = [];
-        const getScreenElementRect = (element) => {
-            const bounds = String(element.bounds);
-            const [, left, top, right, bottom] = bounds.match(/^\[(\d+),(\d+)\]\[(\d+),(\d+)\]$/)?.map(Number) || [];
-            return {
-                x: left,
-                y: top,
-                width: right - left,
-                height: bottom - top,
-            };
-        };
         if (node.node) {
             if (Array.isArray(node.node)) {
                 for (const childNode of node.node) {
@@ -145,10 +156,14 @@ class AndroidRobot {
         if (node.text || node["content-desc"] || node.hint) {
             const element = {
                 type: node.class || "text",
-                name: node.text,
+                text: node.text,
                 label: node["content-desc"] || node.hint || "",
-                rect: getScreenElementRect(node),
+                rect: this.getScreenElementRect(node),
             };
+            if (node.focused === "true") {
+                // only provide it if it's true, otherwise don't confuse llm
+                element.focused = true;
+            }
             if (element.rect.width > 0 && element.rect.height > 0) {
                 elements.push(element);
             }
@@ -156,12 +171,7 @@ class AndroidRobot {
         return elements;
     }
     async getElementsOnScreen() {
-        const dump = this.adb("exec-out", "uiautomator", "dump", "/dev/tty");
-        const parser = new xml.XMLParser({
-            ignoreAttributes: false,
-            attributeNamePrefix: ""
-        });
-        const parsedXml = parser.parse(dump);
+        const parsedXml = await this.getUiAutomatorXml();
         const hierarchy = parsedXml.hierarchy;
         const elements = this.collectElements(hierarchy.node);
         return elements;
@@ -186,14 +196,80 @@ class AndroidRobot {
     async tap(x, y) {
         this.adb("shell", "input", "tap", `${x}`, `${y}`);
     }
+    async setOrientation(orientation) {
+        // Android uses numbers for orientation:
+        // 0 - Portrait
+        // 1 - Landscape
+        const orientationValue = orientation === "portrait" ? 0 : 1;
+        // Set orientation using content provider
+        this.adb("shell", "content", "insert", "--uri", "content://settings/system", "--bind", "name:s:user_rotation", "--bind", `value:i:${orientationValue}`);
+        // Force the orientation change
+        this.adb("shell", "settings", "put", "system", "accelerometer_rotation", "0");
+    }
+    async getOrientation() {
+        const rotation = this.adb("shell", "settings", "get", "system", "user_rotation").toString().trim();
+        return rotation === "0" ? "portrait" : "landscape";
+    }
+    async getUiAutomatorDump() {
+        for (let tries = 0; tries < 10; tries++) {
+            const dump = this.adb("exec-out", "uiautomator", "dump", "/dev/tty").toString();
+            // note: we're not catching other errors here. maybe we should check for <?xml
+            if (dump.includes("null root node returned by UiTestAutomationBridge")) {
+                // uncomment for debugging
+                // const screenshot = await this.getScreenshot();
+                // console.error("Failed to get UIAutomator XML. Here's a screenshot: " + screenshot.toString("base64"));
+                continue;
+            }
+            return dump;
+        }
+        throw new robot_1.ActionableError("Failed to get UIAutomator XML");
+    }
+    async getUiAutomatorXml() {
+        const dump = await this.getUiAutomatorDump();
+        const parser = new xml.XMLParser({
+            ignoreAttributes: false,
+            attributeNamePrefix: ""
+        });
+        return parser.parse(dump);
+    }
+    getScreenElementRect(node) {
+        const bounds = String(node.bounds);
+        const [, left, top, right, bottom] = bounds.match(/^\[(\d+),(\d+)\]\[(\d+),(\d+)\]$/)?.map(Number) || [];
+        return {
+            x: left,
+            y: top,
+            width: right - left,
+            height: bottom - top,
+        };
+    }
 }
 exports.AndroidRobot = AndroidRobot;
-const getConnectedDevices = () => {
-    return (0, child_process_1.execFileSync)(getAdbPath(), ["devices"])
-        .toString()
-        .split("\n")
-        .filter(line => !line.startsWith("List of devices attached"))
-        .filter(line => line.trim() !== "")
-        .map(line => line.split("\t")[0]);
-};
-exports.getConnectedDevices = getConnectedDevices;
+class AndroidDeviceManager {
+    getDeviceType(name) {
+        const device = new AndroidRobot(name);
+        const features = device.getSystemFeatures();
+        if (features.includes("android.software.leanback") || features.includes("android.hardware.type.television")) {
+            return "tv";
+        }
+        return "mobile";
+    }
+    getConnectedDevices() {
+        try {
+            const names = (0, child_process_1.execFileSync)(getAdbPath(), ["devices"])
+                .toString()
+                .split("\n")
+                .filter(line => !line.startsWith("List of devices attached"))
+                .filter(line => line.trim() !== "")
+                .map(line => line.split("\t")[0]);
+            return names.map(name => ({
+                deviceId: name,
+                deviceType: this.getDeviceType(name),
+            }));
+        }
+        catch (error) {
+            console.error("Could not execute adb command, maybe ANDROID_HOME is not set?");
+            return [];
+        }
+    }
+}
+exports.AndroidDeviceManager = AndroidDeviceManager;

package/lib/image-utils.js

@@ -0,0 +1,64 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.isImageMagickInstalled = exports.Image = exports.ImageTransformer = void 0;
+const child_process_1 = require("child_process");
+const DEFAULT_JPEG_QUALITY = 75;
+class ImageTransformer {
+    buffer;
+    newWidth = 0;
+    newFormat = "png";
+    jpegOptions = { quality: DEFAULT_JPEG_QUALITY };
+    constructor(buffer) {
+        this.buffer = buffer;
+    }
+    resize(width) {
+        this.newWidth = width;
+        return this;
+    }
+    jpeg(options) {
+        this.newFormat = "jpg";
+        this.jpegOptions = options;
+        return this;
+    }
+    png() {
+        this.newFormat = "png";
+        return this;
+    }
+    toBuffer() {
+        const proc = (0, child_process_1.spawnSync)("magick", ["-", "-resize", `${this.newWidth}x`, "-quality", `${this.jpegOptions.quality}`, `${this.newFormat}:-`], {
+            maxBuffer: 8 * 1024 * 1024,
+            input: this.buffer
+        });
+        return proc.stdout;
+    }
+}
+exports.ImageTransformer = ImageTransformer;
+class Image {
+    buffer;
+    constructor(buffer) {
+        this.buffer = buffer;
+    }
+    static fromBuffer(buffer) {
+        return new Image(buffer);
+    }
+    resize(width) {
+        return new ImageTransformer(this.buffer).resize(width);
+    }
+    jpeg(options) {
+        return new ImageTransformer(this.buffer).jpeg(options);
+    }
+}
+exports.Image = Image;
+const isImageMagickInstalled = () => {
+    try {
+        return (0, child_process_1.execFileSync)("magick", ["--version"])
+            .toString()
+            .split("\n")
+            .filter(line => line.includes("Version: ImageMagick"))
+            .length > 0;
+    }
+    catch (error) {
+        return false;
+    }
+};
+exports.isImageMagickInstalled = isImageMagickInstalled;

package/lib/ios.js

@@ -132,6 +132,14 @@ class IosRobot {
         (0, fs_1.unlinkSync)(tmpFilename);
         return buffer;
     }
+    async setOrientation(orientation) {
+        const wda = await this.wda();
+        await wda.setOrientation(orientation);
+    }
+    async getOrientation() {
+        const wda = await this.wda();
+        return await wda.getOrientation();
+    }
 }
 exports.IosRobot = IosRobot;
 class IosManager {
@@ -145,6 +153,11 @@ class IosManager {
             return false;
         }
     }
+    async getDeviceName(deviceId) {
+        const output = (0, child_process_1.execFileSync)(getGoIosPath(), ["info", "--udid", deviceId]).toString();
+        const json = JSON.parse(output);
+        return json.DeviceName;
+    }
     async listDevices() {
         if (!(await this.isGoIosInstalled())) {
             console.error("go-ios is not installed, no physical iOS devices can be detected");
@@ -152,7 +165,11 @@ class IosManager {
         }
         const output = (0, child_process_1.execFileSync)(getGoIosPath(), ["list"]).toString();
         const json = JSON.parse(output);
-        return json.deviceList;
+        const devices = json.deviceList.map(async (device) => ({
+            deviceId: device,
+            deviceName: await this.getDeviceName(device),
+        }));
+        return Promise.all(devices);
     }
 }
 exports.IosManager = IosManager;

package/lib/iphone-simulator.js

@@ -15,7 +15,7 @@ class Simctl {
     async wda() {
         const wda = new webdriver_agent_1.WebDriverAgent("localhost", WDA_PORT);
         if (!(await wda.isRunning())) {
-            throw new robot_1.ActionableError("WebDriverAgent is not running on device (tunnel okay, port forwarding okay), please see https://github.com/mobile-next/mobile-mcp/wiki/");
+            throw new robot_1.ActionableError("WebDriverAgent is not running on simulator, please see https://github.com/mobile-next/mobile-mcp/wiki/");
         }
         return wda;
     }
@@ -100,21 +100,40 @@ class Simctl {
         const wda = await this.wda();
         return wda.getElementsOnScreen();
     }
+    async setOrientation(orientation) {
+        const wda = await this.wda();
+        return wda.setOrientation(orientation);
+    }
+    async getOrientation() {
+        const wda = await this.wda();
+        return wda.getOrientation();
+    }
 }
 exports.Simctl = Simctl;
 class SimctlManager {
     listSimulators() {
-        const text = (0, child_process_1.execFileSync)("xcrun", ["simctl", "list", "devices", "-j"]).toString();
-        const json = JSON.parse(text);
-        return Object.values(json.devices).flatMap(device => {
-            return device.map(d => {
-                return {
-                    name: d.name,
-                    uuid: d.udid,
-                    state: d.state,
-                };
+        // detect if this is a mac
+        if (process.platform !== "darwin") {
+            // don't even try to run xcrun
+            return [];
+        }
+        try {
+            const text = (0, child_process_1.execFileSync)("xcrun", ["simctl", "list", "devices", "-j"]).toString();
+            const json = JSON.parse(text);
+            return Object.values(json.devices).flatMap(device => {
+                return device.map(d => {
+                    return {
+                        name: d.name,
+                        uuid: d.udid,
+                        state: d.state,
+                    };
+                });
             });
-        });
+        }
+        catch (error) {
+            console.error("Error listing simulators", error);
+            return [];
+        }
     }
     listBootedSimulators() {
         return this.listSimulators()

package/lib/png.js

@@ -0,0 +1,19 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.PNG = void 0;
+class PNG {
+    buffer;
+    constructor(buffer) {
+        this.buffer = buffer;
+    }
+    getDimensions() {
+        const pngSignature = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);
+        if (!this.buffer.subarray(0, 8).equals(pngSignature)) {
+            throw new Error("Not a valid PNG file");
+        }
+        const width = this.buffer.readUInt32BE(16);
+        const height = this.buffer.readUInt32BE(20);
+        return { width, height };
+    }
+}
+exports.PNG = PNG;

package/lib/server.js

@@ -1,17 +1,15 @@
 "use strict";
-var __importDefault = (this && this.__importDefault) || function (mod) {
-    return (mod && mod.__esModule) ? mod : { "default": mod };
-};
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.createMcpServer = void 0;
 const mcp_js_1 = require("@modelcontextprotocol/sdk/server/mcp.js");
 const zod_1 = require("zod");
-const sharp_1 = __importDefault(require("sharp"));
 const logger_1 = require("./logger");
 const android_1 = require("./android");
 const robot_1 = require("./robot");
 const iphone_simulator_1 = require("./iphone-simulator");
 const ios_1 = require("./ios");
+const png_1 = require("./png");
+const image_utils_1 = require("./image-utils");
 const getAgentVersion = () => {
     const json = require("../package.json");
     return json.version;
@@ -62,11 +60,28 @@ const createMcpServer = () => {
     };
     tool("mobile_list_available_devices", "List all available devices. This includes both physical devices and simulators. If there is more than one device returned, you need to let the user select one of them.", {}, async ({}) => {
         const iosManager = new ios_1.IosManager();
-        const devices = await simulatorManager.listBootedSimulators();
+        const androidManager = new android_1.AndroidDeviceManager();
+        const devices = simulatorManager.listBootedSimulators();
         const simulatorNames = devices.map(d => d.name);
-        const androidDevices = (0, android_1.getConnectedDevices)();
+        const androidDevices = androidManager.getConnectedDevices();
         const iosDevices = await iosManager.listDevices();
-        return `Found these iOS simulators: [${simulatorNames.join(".")}], iOS devices: [${iosDevices.join(",")}] and Android devices: [${androidDevices.join(",")}]`;
+        const iosDeviceNames = iosDevices.map(d => d.deviceId);
+        const androidTvDevices = androidDevices.filter(d => d.deviceType === "tv").map(d => d.deviceId);
+        const androidMobileDevices = androidDevices.filter(d => d.deviceType === "mobile").map(d => d.deviceId);
+        const resp = ["Found these devices:"];
+        if (simulatorNames.length > 0) {
+            resp.push(`iOS simulators: [${simulatorNames.join(".")}]`);
+        }
+        if (iosDevices.length > 0) {
+            resp.push(`iOS devices: [${iosDeviceNames.join(",")}]`);
+        }
+        if (androidMobileDevices.length > 0) {
+            resp.push(`Android devices: [${androidMobileDevices.join(",")}]`);
+        }
+        if (androidTvDevices.length > 0) {
+            resp.push(`Android TV devices: [${androidTvDevices.join(",")}]`);
+        }
+        return resp.join("\n");
     });
     tool("mobile_use_device", "Select a device to use. This can be a simulator or an Android device. Use the list_available_devices tool to get a list of available devices.", {
         device: zod_1.z.string().describe("The name of the device to select"),
@@ -83,7 +98,7 @@ const createMcpServer = () => {
                 robot = new android_1.AndroidRobot(device);
                 break;
         }
-        return `Selected device: ${device} (${deviceType})`;
+        return `Selected device: ${device}`;
     });
     tool("mobile_list_apps", "List all the installed apps on the device", {}, async ({}) => {
         requireRobot();
@@ -121,17 +136,25 @@ const createMcpServer = () => {
         requireRobot();
         const elements = await robot.getElementsOnScreen();
         const result = elements.map(element => {
-            const x = Number((element.rect.x + element.rect.width / 2)).toFixed(3);
-            const y = Number((element.rect.y + element.rect.height / 2)).toFixed(3);
-            return {
-                text: element.label,
+            const x = Number((element.rect.x + element.rect.width / 2)).toFixed(1);
+            const y = Number((element.rect.y + element.rect.height / 2)).toFixed(1);
+            const out = {
+                type: element.type,
+                text: element.text,
+                label: element.label,
+                name: element.name,
+                value: element.value,
                 coordinates: { x, y }
             };
+            if (element.focused) {
+                out.focused = true;
+            }
+            return out;
         });
         return `Found these elements on screen: ${JSON.stringify(result)}`;
     });
     tool("mobile_press_button", "Press a button on device", {
-        button: zod_1.z.string().describe("The button to press. Supported buttons: BACK (android only), HOME, VOLUME_UP, VOLUME_DOWN, ENTER"),
+        button: zod_1.z.string().describe("The button to press. Supported buttons: BACK (android only), HOME, VOLUME_UP, VOLUME_DOWN, ENTER, DPAD_CENTER (android tv only), DPAD_UP (android tv only), DPAD_DOWN (android tv only), DPAD_LEFT (android tv only), DPAD_RIGHT (android tv only)"),
     }, async ({ button }) => {
         requireRobot();
         await robot.pressButton(button);
@@ -165,24 +188,29 @@ const createMcpServer = () => {
     server.tool("mobile_take_screenshot", "Take a screenshot of the mobile device. Use this to understand what's on screen, if you need to press an element that is available through view hierarchy then you must list elements on screen instead. Do not cache this result.", {}, async ({}) => {
         requireRobot();
         try {
-            const screenshot = await robot.getScreenshot();
-            // Scale down the screenshot by 50%
-            const image = (0, sharp_1.default)(screenshot);
-            const metadata = await image.metadata();
-            if (!metadata.width) {
-                throw new Error("Failed to get screenshot metadata");
+            let screenshot = await robot.getScreenshot();
+            let mimeType = "image/png";
+            // validate we received a png, will throw exception otherwise
+            const image = new png_1.PNG(screenshot);
+            const pngSize = image.getDimensions();
+            if (pngSize.width <= 0 || pngSize.height <= 0) {
+                throw new robot_1.ActionableError("Screenshot is invalid. Please try again.");
+            }
+            if ((0, image_utils_1.isImageMagickInstalled)()) {
+                (0, logger_1.trace)("ImageMagick is installed, resizing screenshot");
+                const image = image_utils_1.Image.fromBuffer(screenshot);
+                const beforeSize = screenshot.length;
+                screenshot = image.resize(Math.floor(pngSize.width / 2))
+                    .jpeg({ quality: 75 })
+                    .toBuffer();
+                const afterSize = screenshot.length;
+                (0, logger_1.trace)(`Screenshot resized from ${beforeSize} bytes to ${afterSize} bytes`);
+                mimeType = "image/jpeg";
             }
-            const resizedScreenshot = await image
-                .resize(Math.floor(metadata.width / 2))
-                .jpeg({ quality: 75 })
-                .toBuffer();
-            // debug:
-            // writeFileSync('/tmp/screenshot.png', screenshot);
-            // writeFileSync('/tmp/screenshot-scaled.jpg', resizedScreenshot);
-            const screenshot64 = resizedScreenshot.toString("base64");
+            const screenshot64 = screenshot.toString("base64");
             (0, logger_1.trace)(`Screenshot taken: ${screenshot.length} bytes`);
             return {
-                content: [{ type: "image", data: screenshot64, mimeType: "image/jpeg" }]
+                content: [{ type: "image", data: screenshot64, mimeType }]
             };
         }
         catch (err) {
@@ -193,6 +221,18 @@ const createMcpServer = () => {
             };
         }
     });
+    tool("mobile_set_orientation", "Change the screen orientation of the device", {
+        orientation: zod_1.z.enum(["portrait", "landscape"]).describe("The desired orientation"),
+    }, async ({ orientation }) => {
+        requireRobot();
+        await robot.setOrientation(orientation);
+        return `Changed device orientation to ${orientation}`;
+    });
+    tool("mobile_get_orientation", "Get the current screen orientation of the device", {}, async () => {
+        requireRobot();
+        const orientation = await robot.getOrientation();
+        return `Current device orientation is ${orientation}`;
+    });
     return server;
 };
 exports.createMcpServer = createMcpServer;

package/lib/webdriver-agent.js

@@ -207,5 +207,25 @@ class WebDriverAgent {
             });
         });
     }
+    async setOrientation(orientation) {
+        await this.withinSession(async (sessionUrl) => {
+            const url = `${sessionUrl}/orientation`;
+            await fetch(url, {
+                method: "POST",
+                headers: { "Content-Type": "application/json" },
+                body: JSON.stringify({
+                    orientation: orientation.toUpperCase()
+                })
+            });
+        });
+    }
+    async getOrientation() {
+        return this.withinSession(async (sessionUrl) => {
+            const url = `${sessionUrl}/orientation`;
+            const response = await fetch(url);
+            const json = await response.json();
+            return json.value.toLowerCase();
+        });
+    }
 }
 exports.WebDriverAgent = WebDriverAgent;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mobilenext/mobile-mcp",
-  "version": "0.0.12",
+  "version": "0.0.14",
   "description": "Mobile MCP",
   "repository": {
     "type": "git",
@@ -24,8 +24,6 @@
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.6.1",
     "fast-xml-parser": "^5.0.9",
-    "nyc": "^17.1.0",
-    "sharp": "^0.33.5",
     "zod-to-json-schema": "^3.24.4"
   },
   "devDependencies": {
@@ -42,6 +40,7 @@
     "eslint-plugin-import": "^2.31.0",
     "eslint-plugin-notice": "^1.0.0",
     "husky": "^9.1.7",
+    "nyc": "^17.1.0",
     "mocha": "^11.1.0",
     "ts-node": "^10.9.2",
     "typescript": "^5.8.2"