cua-agent 0.1.21__tar.gz → 0.1.23__tar.gz

{cua_agent-0.1.21 → cua_agent-0.1.23}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cua-agent
-Version: 0.1.21
+Version: 0.1.23
 Summary: CUA (Computer Use) Agent for AI-driven computer interaction
 Author-Email: TryCua <gh@trycua.com>
 Requires-Python: <3.13,>=3.10
@@ -21,6 +21,9 @@ Requires-Dist: boto3<2.0.0,>=1.35.81; extra == "anthropic"
 Provides-Extra: openai
 Requires-Dist: openai<2.0.0,>=1.14.0; extra == "openai"
 Requires-Dist: httpx<0.29.0,>=0.27.0; extra == "openai"
+Provides-Extra: ui
+Requires-Dist: gradio<6.0.0,>=5.23.3; extra == "ui"
+Requires-Dist: python-dotenv<2.0.0,>=1.0.1; extra == "ui"
 Provides-Extra: som
 Requires-Dist: torch>=2.2.1; extra == "som"
 Requires-Dist: torchvision>=0.17.1; extra == "som"
@@ -33,6 +36,19 @@ Requires-Dist: openai<2.0.0,>=1.14.0; extra == "som"
 Requires-Dist: groq<0.5.0,>=0.4.0; extra == "som"
 Requires-Dist: dashscope<2.0.0,>=1.13.0; extra == "som"
 Requires-Dist: requests<3.0.0,>=2.31.0; extra == "som"
+Provides-Extra: omni
+Requires-Dist: torch>=2.2.1; extra == "omni"
+Requires-Dist: torchvision>=0.17.1; extra == "omni"
+Requires-Dist: ultralytics>=8.0.0; extra == "omni"
+Requires-Dist: transformers>=4.38.2; extra == "omni"
+Requires-Dist: cua-som<0.2.0,>=0.1.0; extra == "omni"
+Requires-Dist: anthropic<0.47.0,>=0.46.0; extra == "omni"
+Requires-Dist: boto3<2.0.0,>=1.35.81; extra == "omni"
+Requires-Dist: openai<2.0.0,>=1.14.0; extra == "omni"
+Requires-Dist: groq<0.5.0,>=0.4.0; extra == "omni"
+Requires-Dist: dashscope<2.0.0,>=1.13.0; extra == "omni"
+Requires-Dist: requests<3.0.0,>=2.31.0; extra == "omni"
+Requires-Dist: ollama<0.5.0,>=0.4.7; extra == "omni"
 Provides-Extra: all
 Requires-Dist: torch>=2.2.1; extra == "all"
 Requires-Dist: torchvision>=0.17.1; extra == "all"
@@ -45,6 +61,9 @@ Requires-Dist: openai<2.0.0,>=1.14.0; extra == "all"
 Requires-Dist: groq<0.5.0,>=0.4.0; extra == "all"
 Requires-Dist: dashscope<2.0.0,>=1.13.0; extra == "all"
 Requires-Dist: requests<3.0.0,>=2.31.0; extra == "all"
+Requires-Dist: ollama<0.5.0,>=0.4.7; extra == "all"
+Requires-Dist: gradio<6.0.0,>=5.23.3; extra == "all"
+Requires-Dist: python-dotenv<2.0.0,>=1.0.1; extra == "all"
 Description-Content-Type: text/markdown
 
 <div align="center">
@@ -80,7 +99,8 @@ pip install "cua-agent[all]"
 # or install specific loop providers
 pip install "cua-agent[openai]" # OpenAI Cua Loop
 pip install "cua-agent[anthropic]" # Anthropic Cua Loop
-pip install "cua-agent[omni]" # Cua Loop based on OmniParser
+pip install "cua-agent[omni]" # Cua Loop based on OmniParser (includes Ollama for local models)
+pip install "cua-agent[ui]" # Gradio UI for the agent
 ```
 
 ## Run
@@ -92,6 +112,12 @@ async with Computer() as macos_computer:
       computer=macos_computer,
       loop=AgentLoop.OPENAI,
       model=LLM(provider=LLMProvider.OPENAI)
+      # or
+      # loop=AgentLoop.ANTHROPIC,
+      # model=LLM(provider=LLMProvider.ANTHROPIC)
+      # or
+      # loop=AgentLoop.OMNI,
+      # model=LLM(provider=LLMProvider.OLLAMA, model="gemma3")
   )
 
   tasks = [
@@ -115,6 +141,44 @@ Refer to these notebooks for step-by-step guides on how to use the Computer-Use
 
 - [Agent Notebook](../../notebooks/agent_nb.ipynb) - Complete examples and workflows
 
+## Using the Gradio UI
+
+The agent includes a Gradio-based user interface for easy interaction. To use it:
+
+```bash
+# Install with Gradio support
+pip install "cua-agent[ui]"
+
+# Create a simple launcher script
+```python
+from agent.ui.gradio.app import create_gradio_ui
+
+app = create_gradio_ui()
+app.launch(share=False)
+```
+
+The Gradio UI provides:
+- Selection of different agent loops (OpenAI, Anthropic, OMNI)
+- Model selection for each provider
+- Configuration of agent parameters
+- Chat interface for interacting with the agent
+
+You can also embed the Gradio UI in your own application:
+
+```python
+# Import directly in your application
+from agent.ui.gradio.app import create_gradio_ui
+
+# Create the UI with advanced features
+demo = create_gradio_ui()
+demo.launch()
+
+# Or for a simpler interface
+from agent.ui.gradio import registry
+demo = registry(name='cua:gpt-4o')
+demo.launch()
+```
+
 ## Agent Loops
 
 The `cua-agent` package provides three agent loops variations, based on different CUA models providers and techniques:
@@ -123,7 +187,7 @@ The `cua-agent` package provides three agent loops variations, based on differen
 |:-----------|:-----------------|:------------|:-------------|
 | `AgentLoop.OPENAI` | • `computer_use_preview` | Use OpenAI Operator CUA model | Not Required |
 | `AgentLoop.ANTHROPIC` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219` | Use Anthropic Computer-Use | Not Required |
-| `AgentLoop.OMNI` <br>(experimental) | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
+| `AgentLoop.OMNI` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4`<br>• `phi4`<br>• `phi4-mini`<br>• `gemma3`<br>• `...`<br>• `Any Ollama-compatible model` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
 
 ## AgentResponse
 The `AgentResponse` class represents the structured output returned after each agent turn. It contains the agent's response, reasoning, tool usage, and other metadata. The response format aligns with the new [OpenAI Agent SDK specification](https://platform.openai.com/docs/api-reference/responses) for better consistency across different agent loops.

{cua_agent-0.1.21 → cua_agent-0.1.23}/README.md

@@ -31,7 +31,8 @@ pip install "cua-agent[all]"
 # or install specific loop providers
 pip install "cua-agent[openai]" # OpenAI Cua Loop
 pip install "cua-agent[anthropic]" # Anthropic Cua Loop
-pip install "cua-agent[omni]" # Cua Loop based on OmniParser
+pip install "cua-agent[omni]" # Cua Loop based on OmniParser (includes Ollama for local models)
+pip install "cua-agent[ui]" # Gradio UI for the agent
 ```
 
 ## Run
@@ -43,6 +44,12 @@ async with Computer() as macos_computer:
       computer=macos_computer,
       loop=AgentLoop.OPENAI,
       model=LLM(provider=LLMProvider.OPENAI)
+      # or
+      # loop=AgentLoop.ANTHROPIC,
+      # model=LLM(provider=LLMProvider.ANTHROPIC)
+      # or
+      # loop=AgentLoop.OMNI,
+      # model=LLM(provider=LLMProvider.OLLAMA, model="gemma3")
   )
 
   tasks = [
@@ -66,6 +73,44 @@ Refer to these notebooks for step-by-step guides on how to use the Computer-Use
 
 - [Agent Notebook](../../notebooks/agent_nb.ipynb) - Complete examples and workflows
 
+## Using the Gradio UI
+
+The agent includes a Gradio-based user interface for easy interaction. To use it:
+
+```bash
+# Install with Gradio support
+pip install "cua-agent[ui]"
+
+# Create a simple launcher script
+```python
+from agent.ui.gradio.app import create_gradio_ui
+
+app = create_gradio_ui()
+app.launch(share=False)
+```
+
+The Gradio UI provides:
+- Selection of different agent loops (OpenAI, Anthropic, OMNI)
+- Model selection for each provider
+- Configuration of agent parameters
+- Chat interface for interacting with the agent
+
+You can also embed the Gradio UI in your own application:
+
+```python
+# Import directly in your application
+from agent.ui.gradio.app import create_gradio_ui
+
+# Create the UI with advanced features
+demo = create_gradio_ui()
+demo.launch()
+
+# Or for a simpler interface
+from agent.ui.gradio import registry
+demo = registry(name='cua:gpt-4o')
+demo.launch()
+```
+
 ## Agent Loops
 
 The `cua-agent` package provides three agent loops variations, based on different CUA models providers and techniques:
@@ -74,7 +119,7 @@ The `cua-agent` package provides three agent loops variations, based on differen
 |:-----------|:-----------------|:------------|:-------------|
 | `AgentLoop.OPENAI` | • `computer_use_preview` | Use OpenAI Operator CUA model | Not Required |
 | `AgentLoop.ANTHROPIC` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219` | Use Anthropic Computer-Use | Not Required |
-| `AgentLoop.OMNI` <br>(experimental) | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
+| `AgentLoop.OMNI` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4`<br>• `phi4`<br>• `phi4-mini`<br>• `gemma3`<br>• `...`<br>• `Any Ollama-compatible model` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
 
 ## AgentResponse
 The `AgentResponse` class represents the structured output returned after each agent turn. It contains the agent's response, reasoning, tool usage, and other metadata. The response format aligns with the new [OpenAI Agent SDK specification](https://platform.openai.com/docs/api-reference/responses) for better consistency across different agent loops.

{cua_agent-0.1.21 → cua_agent-0.1.23}/agent/__init__.py

@@ -48,7 +48,7 @@ except Exception as e:
     # Other issues with telemetry
     logger.warning(f"Error initializing telemetry: {e}")
 
-from .providers.omni.types import LLMProvider, LLM
+from .core.types import LLMProvider, LLM
 from .core.factory import AgentLoop
 from .core.agent import ComputerAgent
 

{cua_agent-0.1.21 → cua_agent-0.1.23}/agent/core/agent.py

@@ -6,8 +6,7 @@ import os
 from typing import AsyncGenerator, Optional
 
 from computer import Computer
-from ..providers.omni.types import LLM
-from .. import AgentLoop
+from .types import LLM, AgentLoop
 from .types import AgentResponse
 from .factory import LoopFactory
 from .provider_config import DEFAULT_MODELS, ENV_VARS
@@ -75,6 +74,7 @@ class ComputerAgent:
         # Use the provided LLM object
         self.provider = model.provider
         actual_model_name = model.name or DEFAULT_MODELS.get(self.provider, "")
+        self.provider_base_url = getattr(model, "provider_base_url", None)
 
         # Ensure we have a valid model name
         if not actual_model_name:
@@ -86,7 +86,12 @@ class ComputerAgent:
 
         # Get API key from environment if not provided
         actual_api_key = api_key or os.environ.get(ENV_VARS[self.provider], "")
-        if not actual_api_key:
+        # Ollama and OpenAI-compatible APIs typically don't require an API key
+        if (
+            not actual_api_key
+            and str(self.provider) not in ["ollama", "oaicompat"]
+            and ENV_VARS[self.provider] != "none"
+        ):
             raise ValueError(f"No API key provided for {self.provider}")
 
         # Create the appropriate loop using the factory
@@ -101,6 +106,7 @@ class ComputerAgent:
                 save_trajectory=save_trajectory,
                 trajectory_dir=trajectory_dir,
                 only_n_most_recent_images=only_n_most_recent_images,
+                provider_base_url=self.provider_base_url,
             )
         except ValueError as e:
             logger.error(f"Failed to create loop: {str(e)}")

{cua_agent-0.1.21 → cua_agent-0.1.23}/agent/core/factory.py

@@ -8,10 +8,6 @@ from computer import Computer
 from .types import AgentLoop
 from .base import BaseLoop
 
-# For type checking only
-if TYPE_CHECKING:
-    from ..providers.omni.types import LLMProvider
-
 logger = logging.getLogger(__name__)
 
 
@@ -33,6 +29,7 @@ class LoopFactory:
         trajectory_dir: str = "trajectories",
         only_n_most_recent_images: Optional[int] = None,
         acknowledge_safety_check_callback: Optional[Callable[[str], Awaitable[bool]]] = None,
+        provider_base_url: Optional[str] = None,
     ) -> BaseLoop:
         """Create and return an appropriate loop instance based on type."""
         if loop_type == AgentLoop.ANTHROPIC:
@@ -77,7 +74,7 @@ class LoopFactory:
             try:
                 from ..providers.omni.loop import OmniLoop
                 from ..providers.omni.parser import OmniParser
-                from ..providers.omni.types import LLMProvider
+                from .types import LLMProvider
             except ImportError:
                 raise ImportError(
                     "The 'omni' provider is not installed. "
@@ -99,6 +96,7 @@ class LoopFactory:
                 base_dir=trajectory_dir,
                 only_n_most_recent_images=only_n_most_recent_images,
                 parser=OmniParser(),
+                provider_base_url=provider_base_url,
             )
         else:
             raise ValueError(f"Unsupported loop type: {loop_type}")

{cua_agent-0.1.21 → cua_agent-0.1.23}/agent/core/provider_config.py

@@ -1,15 +1,19 @@
 """Provider-specific configurations and constants."""
 
-from ..providers.omni.types import LLMProvider
+from .types import LLMProvider
 
 # Default models for different providers
 DEFAULT_MODELS = {
     LLMProvider.OPENAI: "gpt-4o",
     LLMProvider.ANTHROPIC: "claude-3-7-sonnet-20250219",
+    LLMProvider.OLLAMA: "gemma3:4b-it-q4_K_M",
+    LLMProvider.OAICOMPAT: "Qwen2.5-VL-7B-Instruct",
 }
 
 # Map providers to their environment variable names
 ENV_VARS = {
     LLMProvider.OPENAI: "OPENAI_API_KEY",
     LLMProvider.ANTHROPIC: "ANTHROPIC_API_KEY",
+    LLMProvider.OLLAMA: "none",
+    LLMProvider.OAICOMPAT: "none",  # OpenAI-compatible API typically doesn't require an API key
 }

cua_agent-0.1.23/agent/core/types.py

@@ -0,0 +1,103 @@
+"""Core type definitions."""
+
+from typing import Any, Dict, List, Optional, TypedDict, Union
+from enum import Enum, StrEnum, auto
+from dataclasses import dataclass
+
+
+class AgentLoop(Enum):
+    """Enumeration of available loop types."""
+
+    ANTHROPIC = auto()  # Anthropic implementation
+    OMNI = auto()  # OmniLoop implementation
+    OPENAI = auto()  # OpenAI implementation
+    OLLAMA = auto()  # OLLAMA implementation
+    # Add more loop types as needed
+
+
+class LLMProvider(StrEnum):
+    """Supported LLM providers."""
+
+    ANTHROPIC = "anthropic"
+    OPENAI = "openai"
+    OLLAMA = "ollama"
+    OAICOMPAT = "oaicompat"
+
+
+@dataclass
+class LLM:
+    """Configuration for LLM model and provider."""
+
+    provider: LLMProvider
+    name: Optional[str] = None
+    provider_base_url: Optional[str] = None
+
+    def __post_init__(self):
+        """Set default model name if not provided."""
+        if self.name is None:
+            from .provider_config import DEFAULT_MODELS
+
+            self.name = DEFAULT_MODELS.get(self.provider)
+
+        # Set default provider URL if none provided
+        if self.provider_base_url is None and self.provider == LLMProvider.OAICOMPAT:
+            # Default for vLLM
+            self.provider_base_url = "http://localhost:8000/v1"
+            # Common alternatives:
+            # - LM Studio: "http://localhost:1234/v1"
+            # - LocalAI: "http://localhost:8080/v1"
+            # - Ollama with OpenAI compatible API: "http://localhost:11434/v1"
+
+
+# For backward compatibility
+LLMModel = LLM
+Model = LLM
+
+
+# Default models for each provider
+PROVIDER_TO_DEFAULT_MODEL: Dict[LLMProvider, str] = {
+    LLMProvider.ANTHROPIC: "claude-3-7-sonnet-20250219",
+    LLMProvider.OPENAI: "gpt-4o",
+    LLMProvider.OLLAMA: "gemma3:4b-it-q4_K_M",
+    LLMProvider.OAICOMPAT: "Qwen2.5-VL-7B-Instruct",
+}
+
+# Environment variable names for each provider
+PROVIDER_TO_ENV_VAR: Dict[LLMProvider, str] = {
+    LLMProvider.ANTHROPIC: "ANTHROPIC_API_KEY",
+    LLMProvider.OPENAI: "OPENAI_API_KEY",
+    LLMProvider.OLLAMA: "none",
+    LLMProvider.OAICOMPAT: "none",
+}
+
+
+class AgentResponse(TypedDict, total=False):
+    """Agent response format."""
+
+    id: str
+    object: str
+    created_at: int
+    status: str
+    error: Optional[str]
+    incomplete_details: Optional[Any]
+    instructions: Optional[Any]
+    max_output_tokens: Optional[int]
+    model: str
+    output: List[Dict[str, Any]]
+    parallel_tool_calls: bool
+    previous_response_id: Optional[str]
+    reasoning: Dict[str, str]
+    store: bool
+    temperature: float
+    text: Dict[str, Dict[str, str]]
+    tool_choice: str
+    tools: List[Dict[str, Union[str, int]]]
+    top_p: float
+    truncation: str
+    usage: Dict[str, Any]
+    user: Optional[str]
+    metadata: Dict[str, Any]
+    response: Dict[str, List[Dict[str, Any]]]
+    # Additional fields for error responses
+    role: str
+    content: Union[str, List[Dict[str, Any]]]

{cua_agent-0.1.21 → cua_agent-0.1.23}/agent/providers/omni/__init__.py

@@ -1,6 +1,6 @@
 """Omni provider implementation."""
 
-from .types import LLMProvider
+from ...core.types import LLMProvider
 from .image_utils import (
     decode_base64_image,
 )

{cua_agent-0.1.21 → cua_agent-0.1.23}/agent/providers/omni/clients/base.py

@@ -1,43 +1,34 @@
 """Base client implementation for Omni providers."""
 
-import os
 import logging
 from typing import Dict, List, Optional, Any, Tuple
-import aiohttp
-import json
 
 logger = logging.getLogger(__name__)
 
+
 class BaseOmniClient:
     """Base class for provider-specific clients."""
-    
-    def __init__(
-        self,
-        api_key: Optional[str] = None,
-        model: Optional[str] = None
-    ):
+
+    def __init__(self, api_key: Optional[str] = None, model: Optional[str] = None):
         """Initialize base client.
-        
+
         Args:
             api_key: Optional API key
             model: Optional model name
         """
         self.api_key = api_key
         self.model = model
-        
+
     async def run_interleaved(
-        self,
-        messages: List[Dict[str, Any]],
-        system: str,
-        max_tokens: Optional[int] = None
+        self, messages: List[Dict[str, Any]], system: str, max_tokens: Optional[int] = None
     ) -> Dict[str, Any]:
         """Run interleaved chat completion.
-        
+
         Args:
             messages: List of message dicts
             system: System prompt
             max_tokens: Optional max tokens override
-            
+
         Returns:
             Response dict
         """

cua_agent-0.1.23/agent/providers/omni/clients/oaicompat.py

@@ -0,0 +1,177 @@
+"""OpenAI-compatible client implementation."""
+
+import os
+import logging
+from typing import Dict, List, Optional, Any
+import aiohttp
+import re
+from .base import BaseOmniClient
+
+logger = logging.getLogger(__name__)
+
+
+# OpenAI-compatible client for the OmniLoop
+class OAICompatClient(BaseOmniClient):
+    """OpenAI-compatible API client implementation.
+
+    This client can be used with any service that implements the OpenAI API protocol, including:
+    - vLLM
+    - LM Studio
+    - LocalAI
+    - Ollama (with OpenAI compatibility)
+    - Text Generation WebUI
+    - Any other service with OpenAI API compatibility
+    """
+
+    def __init__(
+        self,
+        api_key: Optional[str] = None,
+        model: str = "Qwen2.5-VL-7B-Instruct",
+        provider_base_url: Optional[str] = "http://localhost:8000/v1",
+        max_tokens: int = 4096,
+        temperature: float = 0.0,
+    ):
+        """Initialize the OpenAI-compatible client.
+
+        Args:
+            api_key: Not used for local endpoints, usually set to "EMPTY"
+            model: Model name to use
+            provider_base_url: API base URL. Typically in the format "http://localhost:PORT/v1"
+                Examples:
+                - vLLM: "http://localhost:8000/v1"
+                - LM Studio: "http://localhost:1234/v1"
+                - LocalAI: "http://localhost:8080/v1"
+                - Ollama: "http://localhost:11434/v1"
+            max_tokens: Maximum tokens to generate
+            temperature: Generation temperature
+        """
+        super().__init__(api_key="EMPTY", model=model)
+        self.api_key = "EMPTY"  # Local endpoints typically don't require an API key
+        self.model = model
+        self.provider_base_url = (
+            provider_base_url or "http://localhost:8000/v1"
+        )  # Use default if None
+        self.max_tokens = max_tokens
+        self.temperature = temperature
+
+    def _extract_base64_image(self, text: str) -> Optional[str]:
+        """Extract base64 image data from an HTML img tag."""
+        pattern = r'data:image/[^;]+;base64,([^"]+)'
+        match = re.search(pattern, text)
+        return match.group(1) if match else None
+
+    def _get_loggable_messages(self, messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+        """Create a loggable version of messages with image data truncated."""
+        loggable_messages = []
+        for msg in messages:
+            if isinstance(msg.get("content"), list):
+                new_content = []
+                for content in msg["content"]:
+                    if content.get("type") == "image":
+                        new_content.append(
+                            {"type": "image", "image_url": {"url": "[BASE64_IMAGE_DATA]"}}
+                        )
+                    else:
+                        new_content.append(content)
+                loggable_messages.append({"role": msg["role"], "content": new_content})
+            else:
+                loggable_messages.append(msg)
+        return loggable_messages
+
+    async def run_interleaved(
+        self, messages: List[Dict[str, Any]], system: str, max_tokens: Optional[int] = None
+    ) -> Dict[str, Any]:
+        """Run interleaved chat completion.
+
+        Args:
+            messages: List of message dicts
+            system: System prompt
+            max_tokens: Optional max tokens override
+
+        Returns:
+            Response dict
+        """
+        headers = {"Content-Type": "application/json", "Authorization": f"Bearer {self.api_key}"}
+
+        final_messages = [{"role": "system", "content": system}]
+
+        # Process messages
+        for item in messages:
+            if isinstance(item, dict):
+                if isinstance(item["content"], list):
+                    # Content is already in the correct format
+                    final_messages.append(item)
+                else:
+                    # Single string content, check for image
+                    base64_img = self._extract_base64_image(item["content"])
+                    if base64_img:
+                        message = {
+                            "role": item["role"],
+                            "content": [
+                                {
+                                    "type": "image_url",
+                                    "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"},
+                                }
+                            ],
+                        }
+                    else:
+                        message = {
+                            "role": item["role"],
+                            "content": [{"type": "text", "text": item["content"]}],
+                        }
+                    final_messages.append(message)
+            else:
+                # String content, check for image
+                base64_img = self._extract_base64_image(item)
+                if base64_img:
+                    message = {
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "image_url",
+                                "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"},
+                            }
+                        ],
+                    }
+                else:
+                    message = {"role": "user", "content": [{"type": "text", "text": item}]}
+                final_messages.append(message)
+
+        payload = {"model": self.model, "messages": final_messages, "temperature": self.temperature}
+        payload["max_tokens"] = max_tokens or self.max_tokens
+
+        try:
+            async with aiohttp.ClientSession() as session:
+                # Use default base URL if none provided
+                base_url = self.provider_base_url or "http://localhost:8000/v1"
+
+                # Check if the base URL already includes the chat/completions endpoint
+                endpoint_url = base_url
+                if not endpoint_url.endswith("/chat/completions"):
+                    # If the URL ends with /v1, append /chat/completions
+                    if endpoint_url.endswith("/v1"):
+                        endpoint_url = f"{endpoint_url}/chat/completions"
+                    # If the URL doesn't end with /v1, make sure it has a proper structure
+                    elif not endpoint_url.endswith("/"):
+                        endpoint_url = f"{endpoint_url}/chat/completions"
+                    else:
+                        endpoint_url = f"{endpoint_url}chat/completions"
+
+                # Log the endpoint URL for debugging
+                logger.debug(f"Using endpoint URL: {endpoint_url}")
+
+                async with session.post(endpoint_url, headers=headers, json=payload) as response:
+                    response_json = await response.json()
+
+                    if response.status != 200:
+                        error_msg = response_json.get("error", {}).get(
+                            "message", str(response_json)
+                        )
+                        logger.error(f"Error in API call: {error_msg}")
+                        raise Exception(f"API error: {error_msg}")
+
+                    return response_json
+
+        except Exception as e:
+            logger.error(f"Error in API call: {str(e)}")
+            raise