minitap-mobile-use 2.1.0__tar.gz → 2.3.0__tar.gz

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: minitap-mobile-use
-Version: 2.1.0
+Version: 2.3.0
 Summary: AI-powered multi-agent system that automates real Android and iOS devices through low-level control using LangGraph.
 Author: Pierre-Louis Favreau, Jean-Pierre Lo, Nicolas Dehandschoewercker
 License: MIT License
@@ -24,11 +24,11 @@ License: MIT License
          LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
          OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
          SOFTWARE.
-Requires-Dist: langgraph==0.5.0
+Requires-Dist: langgraph>=0.6.6
 Requires-Dist: adbutils==2.9.3
-Requires-Dist: langchain-google-genai==2.1.5
-Requires-Dist: langchain==0.3.26
-Requires-Dist: langchain-core==0.3.66
+Requires-Dist: langchain-google-genai>=2.1.10
+Requires-Dist: langchain>=0.3.27
+Requires-Dist: langchain-core>=0.3.75
 Requires-Dist: jinja2==3.1.6
 Requires-Dist: python-dotenv==1.1.1
 Requires-Dist: pydantic-settings==2.10.1
@@ -42,6 +42,7 @@ Requires-Dist: fastapi==0.111.0
 Requires-Dist: uvicorn[standard]==0.30.1
 Requires-Dist: colorama>=0.4.6
 Requires-Dist: psutil>=5.9.0
+Requires-Dist: langchain-google-vertexai>=2.0.28
 Requires-Dist: ruff==0.5.3 ; extra == 'dev'
 Requires-Dist: pytest==8.4.1 ; extra == 'dev'
 Requires-Dist: pytest-cov==5.0.0 ; extra == 'dev'
@@ -55,7 +56,7 @@ Description-Content-Type: text/markdown
 
 <div align="center">
 
-![mobile-use in Action](./doc/linkedin-demo-with-text.gif)
+![mobile-use in Action](./doc/banner-v2.png)
 
 </div>
 
@@ -69,6 +70,10 @@ Description-Content-Type: text/markdown
     <a href="https://x.com/minitap_ai?t=iRWtI497UhRGLeCKYQekig&s=09"><b>Twitter / X</b></a>
 </p>
 
+[![PyPI version](https://img.shields.io/pypi/v/minitap-mobile-use.svg?color=blue)](https://pypi.org/project/minitap-mobile-use/)
+[![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/minitap-ai/mobile-use/blob/main/LICENSE)
+
 </div>
 
 Mobile-use is a powerful, open-source AI agent that controls your Android or IOS device using natural language. It understands your commands and interacts with the UI to perform tasks, from sending messages to navigating complex apps.
@@ -78,7 +83,7 @@ Mobile-use is a powerful, open-source AI agent that controls your Android or IOS
 ## ✨ Features
 
 - 🗣️ **Natural Language Control**: Interact with your phone using your native language.
-- 📱 **UI-Aware Automation**: Intelligently navigates through app interfaces.
+- 📱 **UI-Aware Automation**: Intelligently navigates through app interfaces (note: currently has limited effectiveness with games as they don't provide accessibility tree data).
 - 📊 **Data Scraping**: Extract information from any app and structure it into your desired format (e.g., JSON) using a natural language description.
 - 🔧 **Extensible & Customizable**: Easily configure different LLMs to power the agents that power mobile-use.
 
@@ -107,11 +112,26 @@ Ready to automate your mobile experience? Follow these steps to get mobile-use u
 
 2.  **(Optional) Customize LLM Configuration:**
     To use different models or providers, create your own LLM configuration file.
+
     ```bash
     cp llm-config.override.template.jsonc llm-config.override.jsonc
     ```
+
     Then, edit `llm-config.override.jsonc` to fit your needs.
 
+    You can also use local LLMs or any other openai-api compatible providers :
+
+    1. Set `OPENAI_BASE_URL` and `OPENAI_API_KEY` in your `.env`
+    2. In your `llm-config.override.jsonc`, set `openai` as the provider for the agent nodes you want, and choose a model supported by your provider.
+
+    > [!NOTE]  
+    > If you want to use Google Vertex AI, you must either:
+    >
+    > - Have credentials configured for your environment (gcloud, workload identity, etc…)
+    > - Store the path to a service account JSON file as the GOOGLE_APPLICATION_CREDENTIALS environment variable
+    >
+    > More information: - [Credential types](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) - [google.auth API reference](https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth)
+
 ### Quick Launch (Docker)
 
 > [!NOTE]  
@@ -132,7 +152,7 @@ Then run in your terminal:
 
 ```bash
 chmod +x mobile-use.sh
-./mobile-use.sh \
+bash ./mobile-use.sh \
   "Open Gmail, find first 3 unread emails, and list their sender and subject line" \
   --output-description "A JSON list of objects, each with 'sender' and 'subject' keys"
 ```

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/README.md

@@ -2,7 +2,7 @@
 
 <div align="center">
 
-![mobile-use in Action](./doc/linkedin-demo-with-text.gif)
+![mobile-use in Action](./doc/banner-v2.png)
 
 </div>
 
@@ -16,6 +16,10 @@
     <a href="https://x.com/minitap_ai?t=iRWtI497UhRGLeCKYQekig&s=09"><b>Twitter / X</b></a>
 </p>
 
+[![PyPI version](https://img.shields.io/pypi/v/minitap-mobile-use.svg?color=blue)](https://pypi.org/project/minitap-mobile-use/)
+[![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/minitap-ai/mobile-use/blob/main/LICENSE)
+
 </div>
 
 Mobile-use is a powerful, open-source AI agent that controls your Android or IOS device using natural language. It understands your commands and interacts with the UI to perform tasks, from sending messages to navigating complex apps.
@@ -25,7 +29,7 @@ Mobile-use is a powerful, open-source AI agent that controls your Android or IOS
 ## ✨ Features
 
 - 🗣️ **Natural Language Control**: Interact with your phone using your native language.
-- 📱 **UI-Aware Automation**: Intelligently navigates through app interfaces.
+- 📱 **UI-Aware Automation**: Intelligently navigates through app interfaces (note: currently has limited effectiveness with games as they don't provide accessibility tree data).
 - 📊 **Data Scraping**: Extract information from any app and structure it into your desired format (e.g., JSON) using a natural language description.
 - 🔧 **Extensible & Customizable**: Easily configure different LLMs to power the agents that power mobile-use.
 
@@ -54,11 +58,26 @@ Ready to automate your mobile experience? Follow these steps to get mobile-use u
 
 2.  **(Optional) Customize LLM Configuration:**
     To use different models or providers, create your own LLM configuration file.
+
     ```bash
     cp llm-config.override.template.jsonc llm-config.override.jsonc
     ```
+
     Then, edit `llm-config.override.jsonc` to fit your needs.
 
+    You can also use local LLMs or any other openai-api compatible providers :
+
+    1. Set `OPENAI_BASE_URL` and `OPENAI_API_KEY` in your `.env`
+    2. In your `llm-config.override.jsonc`, set `openai` as the provider for the agent nodes you want, and choose a model supported by your provider.
+
+    > [!NOTE]  
+    > If you want to use Google Vertex AI, you must either:
+    >
+    > - Have credentials configured for your environment (gcloud, workload identity, etc…)
+    > - Store the path to a service account JSON file as the GOOGLE_APPLICATION_CREDENTIALS environment variable
+    >
+    > More information: - [Credential types](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) - [google.auth API reference](https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth)
+
 ### Quick Launch (Docker)
 
 > [!NOTE]  
@@ -79,7 +98,7 @@ Then run in your terminal:
 
 ```bash
 chmod +x mobile-use.sh
-./mobile-use.sh \
+bash ./mobile-use.sh \
   "Open Gmail, find first 3 unread emails, and list their sender and subject line" \
   --output-description "A JSON list of objects, each with 'sender' and 'subject' keys"
 ```

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/agents/contextor/contextor.py

@@ -1,4 +1,5 @@
 from minitap.mobile_use.agents.executor.utils import is_last_tool_message_take_screenshot
+from minitap.mobile_use.context import MobileUseContext
 from minitap.mobile_use.controllers.mobile_command_controller import get_screen_data
 from minitap.mobile_use.controllers.platform_specific_commands_controller import (
     get_device_date,
@@ -7,7 +8,6 @@ from minitap.mobile_use.controllers.platform_specific_commands_controller import
 from minitap.mobile_use.graph.state import State
 from minitap.mobile_use.utils.decorators import wrap_with_callbacks
 from minitap.mobile_use.utils.logger import get_logger
-from minitap.mobile_use.context import MobileUseContext
 
 logger = get_logger(__name__)
 
@@ -26,7 +26,9 @@ class ContextorNode:
         focused_app_info = get_focused_app_info(self.ctx)
         device_date = get_device_date(self.ctx)
 
-        should_add_screenshot_context = is_last_tool_message_take_screenshot(list(state.messages))
+        should_add_screenshot_context = is_last_tool_message_take_screenshot(
+            list(state.executor_messages)
+        )
 
         return state.sanitize_update(
             ctx=self.ctx,

minitap_mobile_use-2.3.0/minitap/mobile_use/agents/cortex/cortex.md

@@ -0,0 +1,148 @@
+## You are the **Cortex**
+
+Your job is to **analyze the current {{ platform }} mobile device state** and produce **structured decisions** to achieve the current subgoal and more consecutive subgoals if possible.
+
+You must act like a human brain, responsible for giving instructions to your hands (the **Executor** agent). Therefore, you must act with the same imprecision and uncertainty as a human when performing swipe actions: humans don't know where exactly they are swiping (always prefer percentages of width and height instead of absolute coordinates), they just know they are swiping up or down, left or right, and with how much force (usually amplified compared to what's truly needed - go overboard of sliders for instance).
+
+### Core Principle: Break Unproductive Cycles
+
+Your highest priority is to recognize when you are not making progress. You are in an unproductive cycle if a **sequence of actions brings you back to a previous state without achieving the subgoal.**
+
+If you detect a cycle, you are **FORBIDDEN** from repeating it. You must pivot your strategy.
+
+1.  **Announce the Pivot:** In your `agent_thought`, you must briefly state which workflow is failing and what your new approach is.
+
+2.  **Find a Simpler Path:** Abandon the current workflow. Ask yourself: **"How would a human do this if this feature didn't exist?"** This usually means relying on fundamental actions like scrolling, swiping, or navigating through menus manually.
+
+3.  **Retreat as a Last Resort:** If no simpler path exists, declare the subgoal a failure to trigger a replan.
+
+### How to Perceive the Screen: A Two-Sense Approach
+
+To understand the device state, you have two senses, each with its purpose:
+
+1.  **UI Hierarchy (Your sense of "Touch"):**
+    *   **What it is:** A structured list of all elements on the screen.
+    *   **Use it for:** Finding elements by `resource-id`, checking for specific text, and understanding the layout structure.
+    *   **Limitation:** It does NOT tell you what the screen *looks* like. It can be incomplete, and it contains no information about images, colors, or whether an element is visually obscured.
+
+2.  **`glimpse_screen` (Your sense of "Sight"):**
+    *   **What it is:** A tool that provides a real, up-to-date image of the screen.
+    *   **Use it for:** Confirming what is actually visible. This is your source of TRUTH for all visual information (icons, images, element positions, colors).
+    *   **Golden Rule:** When the UI hierarchy is ambiguous, seems incomplete, or when you need to verify a visual detail before acting, **`glimpse_screen` is always the most effective and reliable action.** Never guess what the screen looks like; use your sight to be sure.
+
+  **CRITICAL NOTE ON SIGHT:** The visual information from `glimpse_screen` is **ephemeral**. It is available for **THIS decision turn ONLY**. You MUST extract all necessary information from it IMMEDIATELY, as it will be cleared before the next step.
+### Context You Receive:
+
+- 📱 **Device state**:
+  - Latest **UI hierarchy** and (if available) a **screenshot**.
+  - **CRITICAL NOTE ON SIGHT:** The visual information from `glimpse_screen` is **ephemeral**. It is available for **THIS decision turn ONLY**. You MUST extract all necessary information from it IMMEDIATELY, as it will be cleared before the next step.
+
+- 🧭 **Task context**:
+  - The user's **initial goal**
+  - The **subgoal plan** with their statuses
+  - The **current subgoal** (the one in `PENDING` in the plan)
+  - A list of **agent thoughts** (previous reasoning, observations about the environment)
+  - **Executor agent feedback** on the latest UI decisions
+
+### Your Mission:
+
+Focus on the **current PENDING subgoal and the next subgoals not yet started**.
+
+**CRITICAL: Before making any decision, you MUST thoroughly analyze the agent thoughts history to:**
+- **Detect patterns of failure or repeated attempts** that suggest the current approach isn't working
+- **Identify contradictions** between what was planned and what actually happened
+- **Spot errors in previous reasoning** that need to be corrected
+- **Learn from successful strategies** used in similar situations
+- **Avoid repeating failed approaches** by recognizing when to change strategy
+
+1. **Analyze the agent thoughts first** - Review all previous agent thoughts to understand:
+   - What strategies have been tried and their outcomes
+   - Any errors or misconceptions in previous reasoning
+   - Patterns that indicate success or failure
+   - Whether the current approach should be continued or modified
+
+2. **Then analyze the UI** and environment to understand what action is required, but always in the context of what the agent thoughts reveal about the situation.
+
+3. If some of the subgoals must be **completed** based on your observations, add them to `complete_subgoals_by_ids`. To justify your conclusion, you will fill in the `agent_thought` field based on:
+
+- The current UI state
+- **Critical analysis of past agent thoughts and their accuracy**
+- Recent tool effects and whether they matched expectations from agent thoughts
+- **Any corrections needed to previous reasoning or strategy**
+
+
+### The Rule of Element Interaction
+
+**You MUST follow it for every element interaction.**
+
+When you target a UI element (for a `tap`, `input_text`, `clear_text`, etc.), you **MUST** provide a comprehensive target object containing every piece of information you can find about it.
+
+*   **1. `resource_id`**: Include this if it is present in the UI hierarchy.
+*   **2. `coordinates`**: Include the full bounds (`x`, `y`, `width`, `height`) if they are available.
+*   **3. `text`**: Include the *current text* content of the element (e.g., "Sign In", "Search...", "First Name").
+
+**This is NOT optional.** Providing all three locators if we have, it is the foundation of the system's reliability. It allows next steps to use a fallback mechanism: if the ID fails, it tries the coordinates, etc. Failing to provide this complete context will lead to action failures.
+
+### Outputting Your Decisions
+
+If you decide to act, output a **valid JSON stringified structured set of instructions** for the Executor.
+
+- These must be **concrete low-level actions**.
+- The executor has the following available tools: {{ executor_tools_list }}.
+- Your goal is to achieve subgoals **fast** - so you must put as much actions as possible in your instructions to complete all achievable subgoals (based on your observations) in one go.
+- To open URLs/links directly, use the `open_link` tool - it will automatically handle opening in the appropriate browser. It also handles deep links.
+- When you need to open an app, use the `find_packages` low-level action to try and get its name. Then, simply use the `launch_app` low-level action to launch it.
+-   **Always use a single `input_text` action** to type in a field. This tool handles focusing the element and placing the cursor correctly. If the tool feedback indicates verification is needed or shows None/empty content, perform verification before proceeding.
+- **Only reference UI element IDs or visible texts that are explicitly present in the provided UI hierarchy or screenshot. Do not invent, infer, or guess any IDs or texts that are not directly observed**.
+- **For text clearing**: When you need to completely clear text from an input field, always call the `clear_text` tool with the correct resource_id. This tool automatically focuses the element, and ensures the field is emptied. If you notice this tool fails to clear the text, try to long press the input, select all, and call `erase_one_char`.
+
+### Output
+
+- **complete_subgoals_by_ids** _(optional)_:
+  A list of subgoal IDs that should be marked as completed.
+
+- **Structured Decisions** _(optional)_:
+  A **valid stringified JSON** describing what should be executed **right now** to advance through the subgoals as much as possible.
+
+- **Agent Thought** _(2-4 sentences)_:
+  **MANDATORY: Start by analyzing previous agent thoughts** - Did previous reasoning contain errors? Are we repeating failed approaches? What worked before in similar situations?
+  
+  Then explain your current decision based on this analysis. If there is any information you need to remember for later steps, you must include it here, because only the agent thoughts will be used to produce the final structured output.
+
+  This also helps other agents understand your decision and learn from future failures. **Explicitly mention if you're correcting a previous error or changing strategy based on agent thoughts analysis.**
+  You must also use this field to mention checkpoints when you perform actions without definite ending: for instance "Swiping up to reveal more recipes - last seen recipe was <ID or NAME>, stop when no more".
+
+**Important:** `complete_subgoals_by_ids` and the structured decisions are mutually exclusive: if you provide both, the structured decisions will be ignored. Therefore, you must always prioritize completing subgoals over providing structured decisions.
+
+---
+
+### Example
+
+#### Current Subgoal:
+
+> "Search for Alice in WhatsApp"
+
+#### Structured Decisions:
+
+```text
+"{\"action\": \"tap\", \"target\": {\"text_input_resource_id\": \"com.whatsapp:id/menuitem_search\", \"text_input_coordinates\": {\"x\": 880, \"y\": 150, \"width\": 120, \"height\": 120}, \"text_input_text\": \"Search\"}}"
+```
+
+#### Agent Thought:
+
+> Analysis: No previous attempts, this is a fresh approach. I will tap the search icon to begin searching. I am providing its resource_id, coordinates, and text content to ensure the Executor can find it reliably, following the element rule.
+
+### Input
+
+**Initial Goal:**
+{{ initial_goal }}
+
+**Subgoal Plan:**
+{{ subgoal_plan }}
+
+**Current Subgoal (what needs to be done right now):**
+{{ current_subgoal }}
+
+**Executor agent feedback on latest UI decisions:**
+
+{{ executor_feedback }}

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/agents/cortex/cortex.py

@@ -44,9 +44,8 @@ class CortexNode:
             initial_goal=state.initial_goal,
             subgoal_plan=state.subgoal_plan,
             current_subgoal=get_current_subgoal(state.subgoal_plan),
-            agents_thoughts=state.agents_thoughts,
             executor_feedback=executor_feedback,
-            executor_tools_list=format_tools_list(self.ctx, EXECUTOR_WRAPPERS_TOOLS),
+            executor_tools_list=format_tools_list(ctx=self.ctx, wrappers=EXECUTOR_WRAPPERS_TOOLS),
         )
         messages = [
             SystemMessage(content=system_message),

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/agents/executor/executor.md

@@ -28,7 +28,6 @@ and your previous actions, you must:
 {
   "action": "tap",
   "target": {
-    "text": "Alice",
     "resource_id": "com.whatsapp:id/conversation_item"
   }
 }
@@ -39,7 +38,6 @@ and your previous actions, you must:
 Call the `tap_on_element` tool with:
 
 - `resource_id = "com.whatsapp:id/conversation_item"`
-- `text = "Alice"`
 - `agent_thought = "I'm tapping on the chat item labeled 'Alice' to open the conversation."`
 
 ---
@@ -55,10 +53,14 @@ Call the `tap_on_element` tool with:
 
 When using the `input_text` tool:
 
-- **Always provide the `resource_id` of the element** you want to type into.
+- **Provide all available information** from the following optional parameters to identify the text input element:
+  - `text_input_resource_id`: The resource ID of the text input element (when available)
+  - `text_input_coordinates`: The bounds (ElementBounds) of the text input element (when available)
+  - `text_input_text`: The current text content of the text input element (when available)
+
 - The tool will automatically:
 
-  1. **Focus the element first**
+  1. **Focus the element** using the provided identification parameters
   2. **Move the cursor to the end** of the existing text
   3. **Then type the new text**
 

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/agents/executor/executor.py

@@ -3,6 +3,8 @@ from pathlib import Path
 from jinja2 import Template
 from langchain_core.messages import HumanMessage, SystemMessage
 from langchain_google_genai import ChatGoogleGenerativeAI
+from langchain_google_vertexai.chat_models import ChatVertexAI
+
 from minitap.mobile_use.constants import EXECUTOR_MESSAGES_KEY
 from minitap.mobile_use.context import MobileUseContext
 from minitap.mobile_use.graph.state import State
@@ -56,7 +58,7 @@ class ExecutorNode:
         }
 
         # ChatGoogleGenerativeAI does not support the "parallel_tool_calls" keyword
-        if not isinstance(llm, ChatGoogleGenerativeAI):
+        if not isinstance(llm, ChatGoogleGenerativeAI | ChatVertexAI):
             llm_bind_tools_kwargs["parallel_tool_calls"] = True
 
         llm = llm.bind_tools(**llm_bind_tools_kwargs)

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/agents/executor/utils.py

@@ -1,4 +1,5 @@
 from langchain_core.messages import BaseMessage
+
 from minitap.mobile_use.utils.conversations import is_tool_message
 
 
@@ -7,5 +8,5 @@ def is_last_tool_message_take_screenshot(messages: list[BaseMessage]) -> bool:
         return False
     for msg in messages[::-1]:
         if is_tool_message(msg):
-            return msg.name == "take_screenshot"
+            return msg.name == "glimpse_screen"
     return False

minitap_mobile_use-2.3.0/minitap/mobile_use/agents/outputter/test_outputter.py

@@ -0,0 +1,169 @@
+import sys
+from unittest.mock import AsyncMock, Mock, patch
+
+import pytest
+from pydantic import BaseModel
+
+sys.modules["langgraph.prebuilt.chat_agent_executor"] = Mock()
+sys.modules["minitap.mobile_use.graph.state"] = Mock()
+sys.modules["langchain_google_vertexai"] = Mock()
+sys.modules["langchain_google_genai"] = Mock()
+sys.modules["langchain_openai"] = Mock()
+sys.modules["langchain_cerebras"] = Mock()
+
+from minitap.mobile_use.agents.outputter.outputter import outputter  # noqa: E402
+from minitap.mobile_use.config import LLM, OutputConfig  # noqa: E402
+from minitap.mobile_use.context import MobileUseContext  # noqa: E402
+from minitap.mobile_use.utils.logger import get_logger  # noqa: E402
+
+logger = get_logger(__name__)
+
+
+class MockPydanticSchema(BaseModel):
+    color: str
+    price: float
+    currency_symbol: str
+    website_url: str
+
+
+mock_dict = {
+    "color": "green",
+    "price": 20,
+    "currency_symbol": "$",
+    "website_url": "http://superwebsite.fr",
+}
+
+
+class DummyState:
+    def __init__(self, messages, initial_goal, agents_thoughts):
+        self.messages = messages
+        self.initial_goal = initial_goal
+        self.agents_thoughts = agents_thoughts
+
+
+mocked_state = DummyState(
+    messages=[],
+    initial_goal="Find a green product on my website",
+    agents_thoughts=[
+        "Going on http://superwebsite.fr",
+        "Searching for products",
+        "Filtering by color",
+        "Color 'green' found for a 20 dollars product",
+    ],
+)
+
+
+@pytest.fixture
+def mock_context():
+    """Create a properly mocked context with all required fields."""
+    ctx = Mock(spec=MobileUseContext)
+    ctx.llm_config = {
+        "executor": LLM(provider="openai", model="gpt-5-nano"),
+        "cortex": LLM(provider="openai", model="gpt-5-nano"),
+        "planner": LLM(provider="openai", model="gpt-5-nano"),
+        "orchestrator": LLM(provider="openai", model="gpt-5-nano"),
+    }
+    ctx.device = Mock()
+    ctx.hw_bridge_client = Mock()
+    ctx.screen_api_client = Mock()
+    return ctx
+
+
+@pytest.fixture
+def mock_state():
+    """Create a mock state with test data."""
+    return DummyState(
+        messages=[],
+        initial_goal="Find a green product on my website",
+        agents_thoughts=[
+            "Going on http://superwebsite.fr",
+            "Searching for products",
+            "Filtering by color",
+            "Color 'green' found for a 20 dollars product",
+        ],
+    )
+
+
+@patch("minitap.mobile_use.agents.outputter.outputter.get_llm")
+@pytest.mark.asyncio
+async def test_outputter_with_pydantic_model(mock_get_llm, mock_context, mock_state):
+    """Test outputter with Pydantic model output."""
+    # Mock the structured LLM response
+    mock_structured_llm = AsyncMock()
+    mock_structured_llm.ainvoke.return_value = MockPydanticSchema(
+        color="green", price=20, currency_symbol="$", website_url="http://superwebsite.fr"
+    )
+
+    # Mock the base LLM
+    mock_llm = Mock()
+    mock_llm.with_structured_output.return_value = mock_structured_llm
+    mock_get_llm.return_value = mock_llm
+
+    config = OutputConfig(
+        structured_output=MockPydanticSchema,
+        output_description=None,
+    )
+
+    result = await outputter(ctx=mock_context, output_config=config, graph_output=mock_state)
+
+    assert isinstance(result, dict)
+    assert result.get("color") == "green"
+
+
+@patch("minitap.mobile_use.agents.outputter.outputter.get_llm")
+@pytest.mark.asyncio
+async def test_outputter_with_dict(mock_get_llm, mock_context, mock_state):
+    """Test outputter with dictionary output."""
+    # Mock the structured LLM response for dict
+    mock_structured_llm = AsyncMock()
+    expected_dict = {
+        "color": "green",
+        "price": 20,
+        "currency_symbol": "$",
+        "website_url": "http://superwebsite.fr",
+    }
+    mock_structured_llm.ainvoke.return_value = expected_dict
+
+    # Mock the base LLM
+    mock_llm = Mock()
+    mock_llm.with_structured_output.return_value = mock_structured_llm
+    mock_get_llm.return_value = mock_llm
+
+    config = OutputConfig(
+        structured_output=mock_dict,
+        output_description=None,
+    )
+
+    result = await outputter(ctx=mock_context, output_config=config, graph_output=mock_state)
+
+    assert isinstance(result, dict)
+    assert result.get("color") == "green"
+    assert result.get("price") == 20
+    assert result.get("currency_symbol") == "$"
+    assert result.get("website_url") == "http://superwebsite.fr"
+
+
+@patch("minitap.mobile_use.agents.outputter.outputter.get_llm")
+@pytest.mark.asyncio
+async def test_outputter_with_natural_language_output(mock_get_llm, mock_context, mock_state):
+    """Test outputter with natural language description output."""
+    # Mock the LLM response for natural language output (no structured output)
+    mock_llm = AsyncMock()
+    expected_json = '{"color": "green", "price": 20, "currency_symbol": "$", "website_url": "http://superwebsite.fr"}'
+    mock_llm.ainvoke.return_value = Mock(content=expected_json)
+    mock_get_llm.return_value = mock_llm
+
+    config = OutputConfig(
+        structured_output=None,
+        output_description=(
+            "A JSON object with a color, a price, a currency_symbol and a website_url key"
+        ),
+    )
+
+    result = await outputter(ctx=mock_context, output_config=config, graph_output=mock_state)
+
+    assert isinstance(result, dict)
+    assert result.get("color") == "green"
+    assert result.get("price") == 20
+    assert result.get("currency_symbol") == "$"
+    assert result.get("website_url") == "http://superwebsite.fr"

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/agents/planner/planner.md

@@ -13,7 +13,7 @@ You work like an agile tech lead: defining the key milestones without locking in
    - Don't assume the full UI is visible yet. Plan based on how most mobile apps work, and keep flexibility.
    - List of agents thoughts is empty which is expected, since it is the first plan.
    - Avoid too granular UI actions based tasks (e.g. "tap", "swipe", "copy", "paste") unless explicitly required.
-   - The executor has the following available tools: **{{ executor_tools_list }}**.
+   - The executor has the following available tools: {{ executor_tools_list }}.
      When one of these tools offers a direct shortcut (e.g. `openLink` instead of manually launching a browser and typing a URL), prefer it over decomposed manual steps.
 
 2. **Replanning**

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/agents/planner/planner.py

@@ -30,7 +30,10 @@ class PlannerNode:
 
         system_message = Template(
             Path(__file__).parent.joinpath("planner.md").read_text(encoding="utf-8")
-        ).render(platform=self.ctx.device.mobile_platform.value)
+        ).render(
+            platform=self.ctx.device.mobile_platform.value,
+            executor_tools_list=format_tools_list(ctx=self.ctx, wrappers=EXECUTOR_WRAPPERS_TOOLS),
+        )
         human_message = Template(
             Path(__file__).parent.joinpath("human.md").read_text(encoding="utf-8")
         ).render(
@@ -38,7 +41,6 @@ class PlannerNode:
             initial_goal=state.initial_goal,
             previous_plan="\n".join(str(s) for s in state.subgoal_plan),
             agent_thoughts="\n".join(state.agents_thoughts),
-            executor_tools_list=format_tools_list(self.ctx, EXECUTOR_WRAPPERS_TOOLS),
         )
         messages = [
             SystemMessage(content=system_message),

{minitap_mobile_use-2.1.0 → minitap_mobile_use-2.3.0}/minitap/mobile_use/config.py

@@ -3,7 +3,9 @@ import os
 from pathlib import Path
 from typing import Annotated, Any, Literal
 
+import google.auth
 from dotenv import load_dotenv
+from google.auth.exceptions import DefaultCredentialsError
 from pydantic import BaseModel, Field, SecretStr, ValidationError, model_validator
 from pydantic_settings import BaseSettings
 
@@ -88,7 +90,7 @@ def record_events(output_path: Path | None, events: list[str] | BaseModel | Any)
 
 ### LLM Configuration
 
-LLMProvider = Literal["openai", "google", "openrouter", "xai"]
+LLMProvider = Literal["openai", "google", "openrouter", "xai", "vertexai"]
 LLMUtilsNode = Literal["outputter", "hopper"]
 AgentNode = Literal["planner", "orchestrator", "cortex", "executor"]
 AgentNodeWithFallback = Literal["cortex"]
@@ -98,6 +100,17 @@ DEFAULT_LLM_CONFIG_FILENAME = "llm-config.defaults.jsonc"
 OVERRIDE_LLM_CONFIG_FILENAME = "llm-config.override.jsonc"
 
 
+def validate_vertex_ai_credentials():
+    try:
+        _, project = google.auth.default()
+        if not project:
+            raise Exception("VertexAI requires a Google Cloud project to be set.")
+    except DefaultCredentialsError as e:
+        raise Exception(
+            f"VertexAI requires valid Google Application Default Credentials (ADC): {e}"
+        )
+
+
 class LLM(BaseModel):
     provider: LLMProvider
     model: str
@@ -110,6 +123,8 @@ class LLM(BaseModel):
             case "google":
                 if not settings.GOOGLE_API_KEY:
                     raise Exception(f"{name} requires GOOGLE_API_KEY in .env")
+            case "vertexai":
+                validate_vertex_ai_credentials()
             case "openrouter":
                 if not settings.OPEN_ROUTER_API_KEY:
                     raise Exception(f"{name} requires OPEN_ROUTER_API_KEY in .env")