cua-agent 0.1.29__tar.gz → 0.1.31__tar.gz

{cua_agent-0.1.29 → cua_agent-0.1.31}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cua-agent
-Version: 0.1.29
+Version: 0.1.31
 Summary: CUA (Computer Use) Agent for AI-driven computer interaction
 Author-Email: TryCua <gh@trycua.com>
 Requires-Python: >=3.10
@@ -21,6 +21,8 @@ Requires-Dist: boto3<2.0.0,>=1.35.81; extra == "anthropic"
 Provides-Extra: openai
 Requires-Dist: openai<2.0.0,>=1.14.0; extra == "openai"
 Requires-Dist: httpx<0.29.0,>=0.27.0; extra == "openai"
+Provides-Extra: uitars
+Requires-Dist: httpx<0.29.0,>=0.27.0; extra == "uitars"
 Provides-Extra: ui
 Requires-Dist: gradio<6.0.0,>=5.23.3; extra == "ui"
 Requires-Dist: python-dotenv<2.0.0,>=1.0.1; extra == "ui"
@@ -99,6 +101,7 @@ pip install "cua-agent[all]"
 # or install specific loop providers
 pip install "cua-agent[openai]" # OpenAI Cua Loop
 pip install "cua-agent[anthropic]" # Anthropic Cua Loop
+pip install "cua-agent[uitars]"    # UI-Tars support
 pip install "cua-agent[omni]" # Cua Loop based on OmniParser (includes Ollama for local models)
 pip install "cua-agent[ui]" # Gradio UI for the agent
 ```
@@ -118,6 +121,9 @@ async with Computer() as macos_computer:
       # or
       # loop=AgentLoop.OMNI,
       # model=LLM(provider=LLMProvider.OLLAMA, model="gemma3")
+      # or
+      # loop=AgentLoop.UITARS,
+      # model=LLM(provider=LLMProvider.OAICOMPAT, model="tgi", provider_base_url="https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1")
   )
 
   tasks = [
@@ -143,7 +149,13 @@ Refer to these notebooks for step-by-step guides on how to use the Computer-Use
 
 ## Using the Gradio UI
 
-The agent includes a Gradio-based user interface for easy interaction. To use it:
+The agent includes a Gradio-based user interface for easier interaction.
+
+<div align="center">
+    <img src="../../img/agent_gradio_ui.png"/>
+</div>
+
+To use it:
 
 ```bash
 # Install with Gradio support
@@ -192,6 +204,10 @@ The Gradio UI provides:
 - Configuration of agent parameters
 - Chat interface for interacting with the agent
 
+### Using UI-TARS
+
+You can use UI-TARS by first following the [deployment guide](https://github.com/bytedance/UI-TARS/blob/main/README_deploy.md). This will give you a provider URL like this: `https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1` which you can use in the gradio UI.
+
 ## Agent Loops
 
 The `cua-agent` package provides three agent loops variations, based on different CUA models providers and techniques:
@@ -200,6 +216,7 @@ The `cua-agent` package provides three agent loops variations, based on differen
 |:-----------|:-----------------|:------------|:-------------|
 | `AgentLoop.OPENAI` | • `computer_use_preview` | Use OpenAI Operator CUA model | Not Required |
 | `AgentLoop.ANTHROPIC` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219` | Use Anthropic Computer-Use | Not Required |
+| `AgentLoop.UITARS` | • `ByteDance-Seed/UI-TARS-1.5-7B` | Uses ByteDance's UI-TARS 1.5 model | Not Required |
 | `AgentLoop.OMNI` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4`<br>• `phi4`<br>• `phi4-mini`<br>• `gemma3`<br>• `...`<br>• `Any Ollama or OpenAI-compatible model` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
 
 ## AgentResponse
@@ -241,25 +258,9 @@ async for result in agent.run(task):
           print(output)
 ```
 
-### Gradio UI
-
-You can also interact with the agent using a Gradio interface.
-
-```python
-# Ensure environment variables (e.g., API keys) are loaded
-# You might need a helper function like load_dotenv_files() if using .env
-# from utils import load_dotenv_files
-# load_dotenv_files()
-
-from agent.ui.gradio.app import create_gradio_ui
-
-app = create_gradio_ui()
-app.launch(share=False)
-```
-
 **Note on Settings Persistence:**
 
 *   The Gradio UI automatically saves your configuration (Agent Loop, Model Choice, Custom Base URL, Save Trajectory state, Recent Images count) to a file named `.gradio_settings.json` in the project's root directory when you successfully run a task.
 *   This allows your preferences to persist between sessions.
 *   API keys entered into the custom provider field are **not** saved in this file for security reasons. Manage API keys using environment variables (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) or a `.env` file.
-*   It's recommended to add `.gradio_settings.json` to your `.gitignore` file.
+*   It's recommended to add `.gradio_settings.json` to your `.gitignore` file.

{cua_agent-0.1.29 → cua_agent-0.1.31}/README.md

@@ -31,6 +31,7 @@ pip install "cua-agent[all]"
 # or install specific loop providers
 pip install "cua-agent[openai]" # OpenAI Cua Loop
 pip install "cua-agent[anthropic]" # Anthropic Cua Loop
+pip install "cua-agent[uitars]"    # UI-Tars support
 pip install "cua-agent[omni]" # Cua Loop based on OmniParser (includes Ollama for local models)
 pip install "cua-agent[ui]" # Gradio UI for the agent
 ```
@@ -50,6 +51,9 @@ async with Computer() as macos_computer:
       # or
       # loop=AgentLoop.OMNI,
       # model=LLM(provider=LLMProvider.OLLAMA, model="gemma3")
+      # or
+      # loop=AgentLoop.UITARS,
+      # model=LLM(provider=LLMProvider.OAICOMPAT, model="tgi", provider_base_url="https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1")
   )
 
   tasks = [
@@ -75,7 +79,13 @@ Refer to these notebooks for step-by-step guides on how to use the Computer-Use
 
 ## Using the Gradio UI
 
-The agent includes a Gradio-based user interface for easy interaction. To use it:
+The agent includes a Gradio-based user interface for easier interaction.
+
+<div align="center">
+    <img src="../../img/agent_gradio_ui.png"/>
+</div>
+
+To use it:
 
 ```bash
 # Install with Gradio support
@@ -124,6 +134,10 @@ The Gradio UI provides:
 - Configuration of agent parameters
 - Chat interface for interacting with the agent
 
+### Using UI-TARS
+
+You can use UI-TARS by first following the [deployment guide](https://github.com/bytedance/UI-TARS/blob/main/README_deploy.md). This will give you a provider URL like this: `https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1` which you can use in the gradio UI.
+
 ## Agent Loops
 
 The `cua-agent` package provides three agent loops variations, based on different CUA models providers and techniques:
@@ -132,6 +146,7 @@ The `cua-agent` package provides three agent loops variations, based on differen
 |:-----------|:-----------------|:------------|:-------------|
 | `AgentLoop.OPENAI` | • `computer_use_preview` | Use OpenAI Operator CUA model | Not Required |
 | `AgentLoop.ANTHROPIC` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219` | Use Anthropic Computer-Use | Not Required |
+| `AgentLoop.UITARS` | • `ByteDance-Seed/UI-TARS-1.5-7B` | Uses ByteDance's UI-TARS 1.5 model | Not Required |
 | `AgentLoop.OMNI` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4`<br>• `phi4`<br>• `phi4-mini`<br>• `gemma3`<br>• `...`<br>• `Any Ollama or OpenAI-compatible model` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
 
 ## AgentResponse
@@ -173,25 +188,9 @@ async for result in agent.run(task):
           print(output)
 ```
 
-### Gradio UI
-
-You can also interact with the agent using a Gradio interface.
-
-```python
-# Ensure environment variables (e.g., API keys) are loaded
-# You might need a helper function like load_dotenv_files() if using .env
-# from utils import load_dotenv_files
-# load_dotenv_files()
-
-from agent.ui.gradio.app import create_gradio_ui
-
-app = create_gradio_ui()
-app.launch(share=False)
-```
-
 **Note on Settings Persistence:**
 
 *   The Gradio UI automatically saves your configuration (Agent Loop, Model Choice, Custom Base URL, Save Trajectory state, Recent Images count) to a file named `.gradio_settings.json` in the project's root directory when you successfully run a task.
 *   This allows your preferences to persist between sessions.
 *   API keys entered into the custom provider field are **not** saved in this file for security reasons. Manage API keys using environment variables (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) or a `.env` file.
-*   It's recommended to add `.gradio_settings.json` to your `.gitignore` file.
+*   It's recommended to add `.gradio_settings.json` to your `.gitignore` file.

{cua_agent-0.1.29 → cua_agent-0.1.31}/agent/core/factory.py

@@ -98,5 +98,24 @@ class LoopFactory:
                 parser=OmniParser(),
                 provider_base_url=provider_base_url,
             )
+        elif loop_type == AgentLoop.UITARS:
+            # Lazy import UITARSLoop only when needed
+            try:
+                from ..providers.uitars.loop import UITARSLoop
+            except ImportError:
+                raise ImportError(
+                    "The 'uitars' provider is not installed. "
+                    "Install it with 'pip install cua-agent[all]'"
+                )
+
+            return UITARSLoop(
+                api_key=api_key,
+                model=model_name,
+                computer=computer,
+                save_trajectory=save_trajectory,
+                base_dir=trajectory_dir,
+                only_n_most_recent_images=only_n_most_recent_images,
+                provider_base_url=provider_base_url,
+            )
         else:
             raise ValueError(f"Unsupported loop type: {loop_type}")

{cua_agent-0.1.29 → cua_agent-0.1.31}/agent/core/types.py

@@ -12,6 +12,7 @@ class AgentLoop(Enum):
     OMNI = auto()  # OmniLoop implementation
     OPENAI = auto()  # OpenAI implementation
     OLLAMA = auto()  # OLLAMA implementation
+    UITARS = auto()  # UI-TARS implementation
     # Add more loop types as needed
 
 

{cua_agent-0.1.29 → cua_agent-0.1.31}/agent/providers/omni/clients/oaicompat.py

@@ -93,7 +93,14 @@ class OAICompatClient(BaseOmniClient):
         """
         headers = {"Content-Type": "application/json", "Authorization": f"Bearer {self.api_key}"}
 
-        final_messages = [{"role": "system", "content": system}]
+        final_messages = [
+            {
+                "role": "system", 
+                "content": [
+                    { "type": "text", "text": system }
+                ]
+            }
+        ]
 
         # Process messages
         for item in messages:
@@ -117,7 +124,10 @@ class OAICompatClient(BaseOmniClient):
                     else:
                         message = {
                             "role": item["role"],
-                            "content": [{"type": "text", "text": item["content"]}],
+                            "content": [{
+                                "type": "text", 
+                                "text": item["content"]
+                            }],
                         }
                     final_messages.append(message)
             else:

{cua_agent-0.1.29 → cua_agent-0.1.31}/agent/providers/openai/tools/computer.py

@@ -162,8 +162,8 @@ class ComputerTool(BaseComputerTool, BaseOpenAITool):
                 y = kwargs.get("y")
                 if x is None or y is None:
                     raise ToolError("x and y coordinates are required for scroll action")
-                scroll_x = kwargs.get("scroll_x", 0) // 20
-                scroll_y = kwargs.get("scroll_y", 0) // 20
+                scroll_x = kwargs.get("scroll_x", 0) // 50
+                scroll_y = kwargs.get("scroll_y", 0) // 50
                 return await self.handle_scroll(x, y, scroll_x, scroll_y)
             elif type == "screenshot":
                 return await self.screenshot()
@@ -240,11 +240,7 @@ class ComputerTool(BaseComputerTool, BaseOpenAITool):
 
             if len(mapped_keys) > 1:
                 # For key combinations (like Ctrl+C)
-                for k in mapped_keys:
-                    await self.computer.interface.press_key(k)
-                await asyncio.sleep(0.1)
-                for k in reversed(mapped_keys):
-                    await self.computer.interface.press_key(k)
+                await self.computer.interface.hotkey(*mapped_keys)
             else:
                 # Single key press
                 await self.computer.interface.press_key(mapped_keys[0])

cua_agent-0.1.31/agent/providers/uitars/__init__.py

@@ -0,0 +1 @@
+"""UI-TARS Agent provider package."""

cua_agent-0.1.31/agent/providers/uitars/clients/base.py

@@ -0,0 +1,35 @@
+"""Base client implementation for Omni providers."""
+
+import logging
+from typing import Dict, List, Optional, Any, Tuple
+
+logger = logging.getLogger(__name__)
+
+
+class BaseUITarsClient:
+    """Base class for provider-specific clients."""
+
+    def __init__(self, api_key: Optional[str] = None, model: Optional[str] = None):
+        """Initialize base client.
+
+        Args:
+            api_key: Optional API key
+            model: Optional model name
+        """
+        self.api_key = api_key
+        self.model = model
+
+    async def run_interleaved(
+        self, messages: List[Dict[str, Any]], system: str, max_tokens: Optional[int] = None
+    ) -> Dict[str, Any]:
+        """Run interleaved chat completion.
+
+        Args:
+            messages: List of message dicts
+            system: System prompt
+            max_tokens: Optional max tokens override
+
+        Returns:
+            Response dict
+        """
+        raise NotImplementedError

cua_agent-0.1.31/agent/providers/uitars/clients/oaicompat.py

@@ -0,0 +1,216 @@
+"""OpenAI-compatible client implementation."""
+
+import os
+import logging
+from typing import Dict, List, Optional, Any
+import aiohttp
+import re
+from .base import BaseUITarsClient
+
+logger = logging.getLogger(__name__)
+
+
+# OpenAI-compatible client for the UI_Tars
+class OAICompatClient(BaseUITarsClient):
+    """OpenAI-compatible API client implementation.
+
+    This client can be used with any service that implements the OpenAI API protocol, including:
+    - Huggingface Text Generation Interface endpoints
+    - vLLM
+    - LM Studio
+    - LocalAI
+    - Ollama (with OpenAI compatibility)
+    - Text Generation WebUI
+    - Any other service with OpenAI API compatibility
+    """
+
+    def __init__(
+        self,
+        api_key: Optional[str] = None,
+        model: str = "Qwen2.5-VL-7B-Instruct",
+        provider_base_url: Optional[str] = "http://localhost:8000/v1",
+        max_tokens: int = 4096,
+        temperature: float = 0.0,
+    ):
+        """Initialize the OpenAI-compatible client.
+
+        Args:
+            api_key: Not used for local endpoints, usually set to "EMPTY"
+            model: Model name to use
+            provider_base_url: API base URL. Typically in the format "http://localhost:PORT/v1"
+                Examples:
+                - vLLM: "http://localhost:8000/v1"
+                - LM Studio: "http://localhost:1234/v1"
+                - LocalAI: "http://localhost:8080/v1"
+                - Ollama: "http://localhost:11434/v1"
+            max_tokens: Maximum tokens to generate
+            temperature: Generation temperature
+        """
+        super().__init__(api_key=api_key or "EMPTY", model=model)
+        self.api_key = api_key or "EMPTY" # Local endpoints typically don't require an API key
+        self.model = model
+        self.provider_base_url = (
+            provider_base_url or "http://localhost:8000/v1"
+        )  # Use default if None
+        self.max_tokens = max_tokens
+        self.temperature = temperature
+
+    def _extract_base64_image(self, text: str) -> Optional[str]:
+        """Extract base64 image data from an HTML img tag."""
+        pattern = r'data:image/[^;]+;base64,([^"]+)'
+        match = re.search(pattern, text)
+        return match.group(1) if match else None
+
+    def _get_loggable_messages(self, messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+        """Create a loggable version of messages with image data truncated."""
+        loggable_messages = []
+        for msg in messages:
+            if isinstance(msg.get("content"), list):
+                new_content = []
+                for content in msg["content"]:
+                    if content.get("type") == "image":
+                        new_content.append(
+                            {"type": "image", "image_url": {"url": "[BASE64_IMAGE_DATA]"}}
+                        )
+                    else:
+                        new_content.append(content)
+                loggable_messages.append({"role": msg["role"], "content": new_content})
+            else:
+                loggable_messages.append(msg)
+        return loggable_messages
+
+    async def run_interleaved(
+        self, messages: List[Dict[str, Any]], system: str, max_tokens: Optional[int] = None
+    ) -> Dict[str, Any]:
+        """Run interleaved chat completion.
+
+        Args:
+            messages: List of message dicts
+            system: System prompt
+            max_tokens: Optional max tokens override
+
+        Returns:
+            Response dict
+        """
+        headers = {"Content-Type": "application/json", "Authorization": f"Bearer {self.api_key}"}
+
+        final_messages = [
+            {
+                "role": "system", 
+                "content": [
+                    { "type": "text", "text": system }
+                ]
+            }
+        ]
+        
+        # Process messages
+        for item in messages:
+            if isinstance(item, dict):
+                if isinstance(item["content"], list):
+                    # Content is already in the correct format
+                    final_messages.append(item)
+                else:
+                    # Single string content, check for image
+                    base64_img = self._extract_base64_image(item["content"])
+                    if base64_img:
+                        message = {
+                            "role": item["role"],
+                            "content": [
+                                {
+                                    "type": "image_url",
+                                    "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"},
+                                }
+                            ],
+                        }
+                    else:
+                        message = {
+                            "role": item["role"],
+                            "content": [{"type": "text", "text": item["content"]}],
+                        }
+                    final_messages.append(message)
+            else:
+                # String content, check for image
+                base64_img = self._extract_base64_image(item)
+                if base64_img:
+                    message = {
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "image_url",
+                                "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"},
+                            }
+                        ],
+                    }
+                else:
+                    message = {"role": "user", "content": [{"type": "text", "text": item}]}
+                final_messages.append(message)
+
+        payload = {
+            "model": self.model, 
+            "messages": final_messages, 
+            "max_tokens": max_tokens or self.max_tokens,
+            "temperature": self.temperature,
+            "top_p": 0.7,
+        }
+
+        try:
+            async with aiohttp.ClientSession() as session:
+                # Use default base URL if none provided
+                base_url = self.provider_base_url or "http://localhost:8000/v1"
+
+                # Check if the base URL already includes the chat/completions endpoint
+                
+                endpoint_url = base_url
+                if not endpoint_url.endswith("/chat/completions"):
+                    # If URL is RunPod format, make it OpenAI compatible
+                    if endpoint_url.startswith("https://api.runpod.ai/v2/"):
+                        # Extract RunPod endpoint ID
+                        parts = endpoint_url.split("/")
+                        if len(parts) >= 5:
+                            runpod_id = parts[4]
+                            endpoint_url = f"https://api.runpod.ai/v2/{runpod_id}/openai/v1/chat/completions"
+                    # If the URL ends with /v1, append /chat/completions
+                    elif endpoint_url.endswith("/v1"):
+                        endpoint_url = f"{endpoint_url}/chat/completions"
+                    # If the URL doesn't end with /v1, make sure it has a proper structure
+                    elif not endpoint_url.endswith("/"):
+                        endpoint_url = f"{endpoint_url}/chat/completions"
+                    else:
+                        endpoint_url = f"{endpoint_url}chat/completions"
+
+                # Log the endpoint URL for debugging
+                logger.debug(f"Using endpoint URL: {endpoint_url}")
+
+                async with session.post(endpoint_url, headers=headers, json=payload) as response:
+                    # Log the status and content type
+                    logger.debug(f"Status: {response.status}")
+                    logger.debug(f"Content-Type: {response.headers.get('Content-Type')}")
+                    
+                    # Get the raw text of the response
+                    response_text = await response.text()
+                    logger.debug(f"Response content: {response_text}")
+                    
+                    # Try to parse as JSON if the content type is appropriate
+                    if "application/json" in response.headers.get('Content-Type', ''):
+                        response_json = await response.json()
+                    else:
+                        raise Exception(f"Response is not JSON format")
+                        # # Optionally try to parse it anyway
+                        # try:
+                        #     import json
+                        #     response_json = json.loads(response_text)
+                        # except json.JSONDecodeError as e:
+                        #     print(f"Failed to parse response as JSON: {e}")
+
+                    if response.status != 200:
+                        error_msg = response_json.get("error", {}).get(
+                            "message", str(response_json)
+                        )
+                        logger.error(f"Error in API call: {error_msg}")
+                        raise Exception(f"API error: {error_msg}")
+
+                    return response_json
+
+        except Exception as e:
+            logger.error(f"Error in API call: {str(e)}")
+            raise