cua-agent 0.1.21__py3-none-any.whl → 0.1.23__py3-none-any.whl

{cua_agent-0.1.21.dist-info → cua_agent-0.1.23.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cua-agent
-Version: 0.1.21
+Version: 0.1.23
 Summary: CUA (Computer Use) Agent for AI-driven computer interaction
 Author-Email: TryCua <gh@trycua.com>
 Requires-Python: <3.13,>=3.10
@@ -21,6 +21,9 @@ Requires-Dist: boto3<2.0.0,>=1.35.81; extra == "anthropic"
 Provides-Extra: openai
 Requires-Dist: openai<2.0.0,>=1.14.0; extra == "openai"
 Requires-Dist: httpx<0.29.0,>=0.27.0; extra == "openai"
+Provides-Extra: ui
+Requires-Dist: gradio<6.0.0,>=5.23.3; extra == "ui"
+Requires-Dist: python-dotenv<2.0.0,>=1.0.1; extra == "ui"
 Provides-Extra: som
 Requires-Dist: torch>=2.2.1; extra == "som"
 Requires-Dist: torchvision>=0.17.1; extra == "som"
@@ -33,6 +36,19 @@ Requires-Dist: openai<2.0.0,>=1.14.0; extra == "som"
 Requires-Dist: groq<0.5.0,>=0.4.0; extra == "som"
 Requires-Dist: dashscope<2.0.0,>=1.13.0; extra == "som"
 Requires-Dist: requests<3.0.0,>=2.31.0; extra == "som"
+Provides-Extra: omni
+Requires-Dist: torch>=2.2.1; extra == "omni"
+Requires-Dist: torchvision>=0.17.1; extra == "omni"
+Requires-Dist: ultralytics>=8.0.0; extra == "omni"
+Requires-Dist: transformers>=4.38.2; extra == "omni"
+Requires-Dist: cua-som<0.2.0,>=0.1.0; extra == "omni"
+Requires-Dist: anthropic<0.47.0,>=0.46.0; extra == "omni"
+Requires-Dist: boto3<2.0.0,>=1.35.81; extra == "omni"
+Requires-Dist: openai<2.0.0,>=1.14.0; extra == "omni"
+Requires-Dist: groq<0.5.0,>=0.4.0; extra == "omni"
+Requires-Dist: dashscope<2.0.0,>=1.13.0; extra == "omni"
+Requires-Dist: requests<3.0.0,>=2.31.0; extra == "omni"
+Requires-Dist: ollama<0.5.0,>=0.4.7; extra == "omni"
 Provides-Extra: all
 Requires-Dist: torch>=2.2.1; extra == "all"
 Requires-Dist: torchvision>=0.17.1; extra == "all"
@@ -45,6 +61,9 @@ Requires-Dist: openai<2.0.0,>=1.14.0; extra == "all"
 Requires-Dist: groq<0.5.0,>=0.4.0; extra == "all"
 Requires-Dist: dashscope<2.0.0,>=1.13.0; extra == "all"
 Requires-Dist: requests<3.0.0,>=2.31.0; extra == "all"
+Requires-Dist: ollama<0.5.0,>=0.4.7; extra == "all"
+Requires-Dist: gradio<6.0.0,>=5.23.3; extra == "all"
+Requires-Dist: python-dotenv<2.0.0,>=1.0.1; extra == "all"
 Description-Content-Type: text/markdown
 
 <div align="center">
@@ -80,7 +99,8 @@ pip install "cua-agent[all]"
 # or install specific loop providers
 pip install "cua-agent[openai]" # OpenAI Cua Loop
 pip install "cua-agent[anthropic]" # Anthropic Cua Loop
-pip install "cua-agent[omni]" # Cua Loop based on OmniParser
+pip install "cua-agent[omni]" # Cua Loop based on OmniParser (includes Ollama for local models)
+pip install "cua-agent[ui]" # Gradio UI for the agent
 ```
 
 ## Run
@@ -92,6 +112,12 @@ async with Computer() as macos_computer:
       computer=macos_computer,
       loop=AgentLoop.OPENAI,
       model=LLM(provider=LLMProvider.OPENAI)
+      # or
+      # loop=AgentLoop.ANTHROPIC,
+      # model=LLM(provider=LLMProvider.ANTHROPIC)
+      # or
+      # loop=AgentLoop.OMNI,
+      # model=LLM(provider=LLMProvider.OLLAMA, model="gemma3")
   )
 
   tasks = [
@@ -115,6 +141,44 @@ Refer to these notebooks for step-by-step guides on how to use the Computer-Use
 
 - [Agent Notebook](../../notebooks/agent_nb.ipynb) - Complete examples and workflows
 
+## Using the Gradio UI
+
+The agent includes a Gradio-based user interface for easy interaction. To use it:
+
+```bash
+# Install with Gradio support
+pip install "cua-agent[ui]"
+
+# Create a simple launcher script
+```python
+from agent.ui.gradio.app import create_gradio_ui
+
+app = create_gradio_ui()
+app.launch(share=False)
+```
+
+The Gradio UI provides:
+- Selection of different agent loops (OpenAI, Anthropic, OMNI)
+- Model selection for each provider
+- Configuration of agent parameters
+- Chat interface for interacting with the agent
+
+You can also embed the Gradio UI in your own application:
+
+```python
+# Import directly in your application
+from agent.ui.gradio.app import create_gradio_ui
+
+# Create the UI with advanced features
+demo = create_gradio_ui()
+demo.launch()
+
+# Or for a simpler interface
+from agent.ui.gradio import registry
+demo = registry(name='cua:gpt-4o')
+demo.launch()
+```
+
 ## Agent Loops
 
 The `cua-agent` package provides three agent loops variations, based on different CUA models providers and techniques:
@@ -123,7 +187,7 @@ The `cua-agent` package provides three agent loops variations, based on differen
 |:-----------|:-----------------|:------------|:-------------|
 | `AgentLoop.OPENAI` | • `computer_use_preview` | Use OpenAI Operator CUA model | Not Required |
 | `AgentLoop.ANTHROPIC` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219` | Use Anthropic Computer-Use | Not Required |
-| `AgentLoop.OMNI` <br>(experimental) | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
+| `AgentLoop.OMNI` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4`<br>• `phi4`<br>• `phi4-mini`<br>• `gemma3`<br>• `...`<br>• `Any Ollama-compatible model` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
 
 ## AgentResponse
 The `AgentResponse` class represents the structured output returned after each agent turn. It contains the agent's response, reasoning, tool usage, and other metadata. The response format aligns with the new [OpenAI Agent SDK specification](https://platform.openai.com/docs/api-reference/responses) for better consistency across different agent loops.

{cua_agent-0.1.21.dist-info → cua_agent-0.1.23.dist-info}/RECORD

@@ -1,13 +1,12 @@
-agent/__init__.py,sha256=ZOK-dcYN3o9CQ9XnUipkDR7YKQNNZ3HL_7MJbqHY_-c,1494
-agent/core/README.md,sha256=VOXNVbR0ugxf9gCXYmZtUU2kngZhfi29haT_oSxK0Lk,3559
+agent/__init__.py,sha256=guFGtorDBF6R5hVep0Bvci3_sUJfBlcsq9ss5Kwrej8,1484
 agent/core/__init__.py,sha256=7DhJ_6KKooM6uTmDIlumCnd7OFcU67BYIIR1dpIYUB0,506
-agent/core/agent.py,sha256=WutrveYndS-YBy9BVu1I-Eyj-7AAk1CgACkn5OdWvhU,8010
+agent/core/agent.py,sha256=HUfBe7Uam3TObAmf6KH0GDKuNCNunNmmMcuxS7aZg0Q,8332
 agent/core/base.py,sha256=EoutyMJ2kSJ72Di8KVRiUXc0ZJ1OkA0e7Ej14Y3F87w,7124
 agent/core/callbacks.py,sha256=VbGIf5QkHh3Q0KsLM6wv7hRdIA5WExTVYLm64bckyUA,4306
 agent/core/experiment.py,sha256=Ywj6q3JZFDKicfPuQsDl0vSN55HS7-Cnk3u3EcUCKe8,8866
-agent/core/factory.py,sha256=sy7k323ZPDf8T7UEKNSV9wT2OylckliaPr8KyYhMAi0,3829
+agent/core/factory.py,sha256=rGlSQDjcm61hNLxe9jLZvmMwPypYatNq25yf_SqUghU,3820
 agent/core/messages.py,sha256=-OVMDqcxK5MUHPEkHliK29XFJYMRAc1keFvzrUyrOmM,16231
-agent/core/provider_config.py,sha256=GndsvFdFQHNlxOSE6XcxRAZcyOJmLQ_jSOd66QY0uyk,414
+agent/core/provider_config.py,sha256=Hr9kDFSXdPeqC6hbid3OTykNF0-XVi0wzZyd44a7kww,627
 agent/core/telemetry.py,sha256=HElPd32k_w2SJ6t-Cc3j_2-AKdLbFwh2YlM8QViDgRw,4790
 agent/core/tools.py,sha256=Jes2CFCFqC727WWHbO-sG7V03rBHnQe5X7Oi9ZkuScI,877
 agent/core/tools/__init__.py,sha256=xZen-PqUp2dUaMEHJowXCQm33_5Sxhsx9PSoD0rq6tI,489
@@ -17,7 +16,7 @@ agent/core/tools/collection.py,sha256=NuwTn6dXSyznxWodfmFDQwUlxxaGb4oBPym4AEJABS
 agent/core/tools/computer.py,sha256=lT_aW3huoYpcM8kffuokELupSz_WZG_qkaW1gITRC58,3892
 agent/core/tools/edit.py,sha256=kv4jTKCM0VXrnoNErf7mT-xlr81-7T8v49_VA9y_L4Y,2005
 agent/core/tools/manager.py,sha256=IRsCXjGc076nncQuyIjODoafnHTDhrf9sP5B4q5Pcdo,1742
-agent/core/types.py,sha256=fSOrbGVjidfrTA1XEy7k5uAcq3Ir1sMNXojs8AS0B1Y,1202
+agent/core/types.py,sha256=4XnjuCkZAeyOidqixHp3pWVVf3pxc2l-0hNoYlB3Mrk,2914
 agent/core/visualization.py,sha256=1DuFF5sSeSf5BRSevBMDxml9-ajl7BQLFm5KBUwMbI8,6573
 agent/providers/__init__.py,sha256=b4tIBAaIB1V7p8V0BWipHVnMhfHH_OuVgP4OWGSHdD8,194
 agent/providers/anthropic/__init__.py,sha256=Mj11IZnVshZ2iHkvg4Z5-jrQIaD1WvzDz2Zk_pMwqIA,149
@@ -39,22 +38,23 @@ agent/providers/anthropic/tools/manager.py,sha256=yNvgTkfEqnOz5isDF0RxvmBMZB0uh2
 agent/providers/anthropic/tools/run.py,sha256=xhXdnBK1di9muaO44CEirL9hpGy3NmKbjfMpyeVmn8Y,1595
 agent/providers/anthropic/types.py,sha256=SF00kOMC1ui8j9Ah56KaeiR2cL394qCHjFIsBpXxt5w,421
 agent/providers/anthropic/utils.py,sha256=qDp0bFGQhK1dG9U461iaeCiyoVUsksXmD43g9cedRW8,14367
-agent/providers/omni/__init__.py,sha256=59Eqpr3Nc3EE61VirUkfecAnQuGELdg0t44q5tg3SW8,172
+agent/providers/omni/__init__.py,sha256=5ix67iJdtQNGuGJEjEOF65PwFWO7vdo1QlXD28bRbW4,179
 agent/providers/omni/api_handler.py,sha256=7CpD43lYAqTyNKWfrD8XcM9ekbajqKCTH9p0TWtEQyg,1163
 agent/providers/omni/clients/anthropic.py,sha256=nC_lj3UwrLqx9TIew58yxLqKwrH1_LwJD6EqVSEfp3g,3670
-agent/providers/omni/clients/base.py,sha256=zAAgPi0jl3SWPC730R9l79E8bfYPSo39UtCSE-mrK6I,1076
-agent/providers/omni/clients/openai.py,sha256=E4TAXMUFoYTunJETCWCNx5XAc6xutiN4rB6PlVpzC5s,5972
+agent/providers/omni/clients/base.py,sha256=6lN86XKZT3cgBT9EQdz2akKoqbIvc-NXXIOkYKwXObE,946
+agent/providers/omni/clients/oaicompat.py,sha256=KHFyOBttNUlxJdOEqrR7sS0S-S0LjMz7EVbTIkXyD_Y,7241
+agent/providers/omni/clients/ollama.py,sha256=PmR5EhU9Mi43_o5mZN36XcpiGKp5HbQwlXpiRF9gO3I,4174
+agent/providers/omni/clients/openai.py,sha256=iTSYWEJEM8INFPGJMiUVs8rFn0781XF_ofRkd7NT3gk,5920
 agent/providers/omni/clients/utils.py,sha256=Ani9CVVBm_J2Dl51WG6p1GVuoI6cq8scISrG0pmQ37o,688
 agent/providers/omni/image_utils.py,sha256=wejhWb36yqedsPnLFTFwk2wth8a6txfVWSg4EaNrRdA,908
-agent/providers/omni/loop.py,sha256=6521KV1hBdxnovzT0wfUcYiRCe7agx2TCC0hgQA3TRw,37459
+agent/providers/omni/loop.py,sha256=h9c-Ie4MA84H3XKYiAKA6J4Tec3_ACYxmU--eRuiS8A,39591
 agent/providers/omni/parser.py,sha256=REpQwlwvY1z_N8wbMj6GhOeTiiWVWHhVja_LOxgzbks,11734
 agent/providers/omni/prompts.py,sha256=Mupjy0bUwBjcAeLXpE1r1jisYPSlhwsp-IXJKEKrEtw,3779
 agent/providers/omni/tools/__init__.py,sha256=IC1cMEDoR2ljGcNNthzBRF_VtnDbRL5qvHJWErtNp98,774
 agent/providers/omni/tools/base.py,sha256=HiQ8dp9NbFGlGopbE1wxo0ZbujA7bzCGjCg4tl2lnPE,824
 agent/providers/omni/tools/bash.py,sha256=wocYvWwoaVjHba19CVqc3bvwj8_1qwqYjNaPBbMRlWA,2241
 agent/providers/omni/tools/computer.py,sha256=cB5PrhPmk6acKSENIvzw4rdpjeWx4HQHfSxBLGHzGRE,6964
-agent/providers/omni/tools/manager.py,sha256=GVLudHNpOQnl6aA_IOvqAEMDoKW62ozMZuwst6Z1Hco,2094
-agent/providers/omni/types.py,sha256=EoEiqtW98R2ZlhZb1-II4t1Ctf7qCOG6rUn_uQUHNdM,1021
+agent/providers/omni/tools/manager.py,sha256=UhtasaxGcmkxtz-bP1UJ1a4xdYnD3Cv8PbtB0n2QCDg,2101
 agent/providers/omni/utils.py,sha256=Ikp6ONL1HO637o3KDtv5yv6q-4uIWAzMSQDvGetWXC8,8724
 agent/providers/openai/__init__.py,sha256=8DS6YNZp42NLCacwXsfRaghyczaOCVovX8TgzXUZf_o,165
 agent/providers/openai/api_handler.py,sha256=L1K56dR1j4JsX1sX4OFYeKoCUMM25Fwj2y9nqv8oOhw,17736
@@ -67,7 +67,10 @@ agent/providers/openai/tools/manager.py,sha256=-wM641dLf8vcv6QF9x_ViGJeDl2YTuUV9
 agent/providers/openai/types.py,sha256=0mFUxeFy23fJhMwc6lAFVXKngg2fJIXkPS5oV284V1M,898
 agent/providers/openai/utils.py,sha256=YeCZWIqOFSeugWoqAS0rhxOKAfL-9uN9nrYSBGBgPdc,3175
 agent/telemetry.py,sha256=pVGxbj0ewnvq4EGj28CydN4a1iOfvZR_XKL3vIOqhOM,390
-cua_agent-0.1.21.dist-info/METADATA,sha256=3l3sevJQsLq-kFjOEK5yMztKcWIzuXHI_acK6zbIjTs,7032
-cua_agent-0.1.21.dist-info/WHEEL,sha256=thaaA2w1JzcGC48WYufAs8nrYZjJm8LqNfnXFOFyCC4,90
-cua_agent-0.1.21.dist-info/entry_points.txt,sha256=6OYgBcLyFCUgeqLgnvMyOJxPCWzgy7se4rLPKtNonMs,34
-cua_agent-0.1.21.dist-info/RECORD,,
+agent/ui/__init__.py,sha256=ohhxJLBin6k1hl5sKcmBST8mgh23WXgAXz3pN4f470E,45
+agent/ui/gradio/__init__.py,sha256=ANKZhv1HqsLheWbLVBlyRQ7Q5qGeXuPi5jDs8vu-ZMo,579
+agent/ui/gradio/app.py,sha256=6n0c_3HBb6ZeN213izyurL8oML1peet1cI8fx82DLZg,33980
+cua_agent-0.1.23.dist-info/METADATA,sha256=88aLbVo6etPVlHUPYmxmOpCTRfmeIJ1axKfsrznGG10,9238
+cua_agent-0.1.23.dist-info/WHEEL,sha256=thaaA2w1JzcGC48WYufAs8nrYZjJm8LqNfnXFOFyCC4,90
+cua_agent-0.1.23.dist-info/entry_points.txt,sha256=6OYgBcLyFCUgeqLgnvMyOJxPCWzgy7se4rLPKtNonMs,34
+cua_agent-0.1.23.dist-info/RECORD,,

agent/core/README.md

@@ -1,101 +0,0 @@
-# Unified ComputerAgent
-
-The `ComputerAgent` class provides a unified implementation that consolidates the previously separate agent implementations (AnthropicComputerAgent and OmniComputerAgent) into a single, configurable class.
-
-## Features
-
-- **Multiple Loop Types**: Switch between different agentic loop implementations using the `loop_type` parameter (Anthropic or Omni).
-- **Provider Support**: Use different AI providers (OpenAI, Anthropic, etc.) with the appropriate loop.
-- **Trajectory Saving**: Control whether to save screenshots and logs with the `save_trajectory` parameter.
-- **Consistent Interface**: Maintains a consistent interface regardless of the underlying loop implementation.
-
-## API Key Requirements
-
-To use the ComputerAgent, you'll need API keys for the providers you want to use:
-
-- For **OpenAI**: Set the `OPENAI_API_KEY` environment variable or pass it directly as `api_key`.
-- For **Anthropic**: Set the `ANTHROPIC_API_KEY` environment variable or pass it directly as `api_key`.
-- For **Groq**: Set the `GROQ_API_KEY` environment variable or pass it directly as `api_key`.
-
-You can set environment variables in several ways:
-
-```bash
-# In your terminal before running the code
-export OPENAI_API_KEY=your_api_key_here
-
-# Or in a .env file
-OPENAI_API_KEY=your_api_key_here
-```
-
-## Usage
-
-Here's how to use the unified ComputerAgent:
-
-```python
-from agent.core.agent import ComputerAgent
-from agent.types.base import AgenticLoop
-from agent.providers.omni.types import LLMProvider
-from computer import Computer
-
-# Create a Computer instance
-computer = Computer()
-
-# Create an agent with the OMNI loop and OpenAI provider
-agent = ComputerAgent(
-    computer=computer,
-    loop_type=AgenticLoop.OMNI,
-    provider=LLMProvider.OPENAI,
-    model="gpt-4o",
-    api_key="your_api_key_here",  # Can also use OPENAI_API_KEY environment variable
-    save_trajectory=True,
-    only_n_most_recent_images=5
-)
-
-# Create an agent with the ANTHROPIC loop
-agent = ComputerAgent(
-    computer=computer,
-    loop_type=AgenticLoop.ANTHROPIC,
-    model="claude-3-7-sonnet-20250219",
-    api_key="your_api_key_here",  # Can also use ANTHROPIC_API_KEY environment variable
-    save_trajectory=True,
-    only_n_most_recent_images=5
-)
-
-# Use the agent
-async with agent:
-    async for result in agent.run("Your task description here"):
-        # Process the result
-        title = result["metadata"].get("title", "Screen Analysis")
-        content = result["content"]
-        print(f"\n{title}")
-        print(content)
-```
-
-## Parameters
-
-- `computer`: Computer instance to control
-- `loop_type`: The type of loop to use (AgenticLoop.ANTHROPIC or AgenticLoop.OMNI)
-- `provider`: AI provider to use (required for Omni loop)
-- `api_key`: Optional API key (will use environment variable if not provided)
-- `model`: Optional model name (will use provider default if not specified)
-- `save_trajectory`: Whether to save screenshots and logs
-- `only_n_most_recent_images`: Only keep N most recent images
-- `max_retries`: Maximum number of retry attempts
-
-## Directory Structure
-
-When `save_trajectory` is enabled, the agent will create the following directory structure:
-
-```
-experiments/
-  ├── screenshots/   # Screenshots captured during agent execution
-  └── logs/          # API call logs and other logging information
-```
-
-## Extending with New Loop Types
-
-To add a new loop type:
-
-1. Implement a new loop class
-2. Add a new value to the `AgenticLoop` enum
-3. Update the `_initialize_loop` method in `ComputerAgent` to handle the new loop type 

agent/providers/omni/types.py

@@ -1,44 +0,0 @@
-"""Type definitions for the Omni provider."""
-
-from enum import StrEnum
-from typing import Dict, Optional
-from dataclasses import dataclass
-
-
-class LLMProvider(StrEnum):
-    """Supported LLM providers."""
-
-    ANTHROPIC = "anthropic"
-    OMNI = "omni"
-    OPENAI = "openai"
-
-
-@dataclass
-class LLM:
-    """Configuration for LLM model and provider."""
-
-    provider: LLMProvider
-    name: Optional[str] = None
-
-    def __post_init__(self):
-        """Set default model name if not provided."""
-        if self.name is None:
-            self.name = PROVIDER_TO_DEFAULT_MODEL.get(self.provider)
-
-
-# For backward compatibility
-LLMModel = LLM
-Model = LLM
-
-
-# Default models for each provider
-PROVIDER_TO_DEFAULT_MODEL: Dict[LLMProvider, str] = {
-    LLMProvider.ANTHROPIC: "claude-3-7-sonnet-20250219",
-    LLMProvider.OPENAI: "gpt-4o",
-}
-
-# Environment variable names for each provider
-PROVIDER_TO_ENV_VAR: Dict[LLMProvider, str] = {
-    LLMProvider.ANTHROPIC: "ANTHROPIC_API_KEY",
-    LLMProvider.OPENAI: "OPENAI_API_KEY",
-}