cognitrix 0.2.0__tar.gz → 0.2.2__tar.gz

{cognitrix-0.2.0 → cognitrix-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cognitrix
-Version: 0.2.0
+Version: 0.2.2
 Summary: Package for creating AI Agents using llms
 Home-page: https://github.com/theonlyamos/cognitrix
 License: Apache2
@@ -23,10 +23,13 @@ Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Topic :: Software Development :: Build Tools
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Dist: aiofiles (>=23.2.1,<24.0.0)
 Requires-Dist: aiohttp (>=3.9.3,<4.0.0)
 Requires-Dist: anthropic (>=0.17.0,<0.18.0)
+Requires-Dist: beautifulsoup4 (>=4.12.3,<5.0.0)
 Requires-Dist: clarifai (>=10.1.0,<11.0.0)
 Requires-Dist: cohere (>=4.50,<5.0)
+Requires-Dist: deepgram-sdk (==3.*)
 Requires-Dist: google-generativeai (>=0.4.0,<0.5.0)
 Requires-Dist: google-search-results (>=2.4.2,<3.0.0)
 Requires-Dist: groq (>=0.4.1,<0.5.0)
@@ -35,10 +38,12 @@ Requires-Dist: pillow (>=10.2.0,<11.0.0)
 Requires-Dist: pyautogui (>=0.9.54,<0.10.0)
 Requires-Dist: pydantic (>=2.6.2,<3.0.0)
 Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Requires-Dist: pyttsx3 (>=2.90,<3.0)
 Requires-Dist: requests (>=2.31.0,<3.0.0)
 Requires-Dist: serpapi (>=0.1.5,<0.2.0)
 Requires-Dist: together (>=0.2.11,<0.3.0)
 Requires-Dist: uuid (>=1.30,<2.0)
+Requires-Dist: xmltodict (>=0.13.0,<0.14.0)
 Project-URL: Bug Tracker, https://github.com/theonlyamos/cognitrix/issues
 Project-URL: Repository, https://github.com/theonlyamos/cognitrix
 Description-Content-Type: text/markdown
@@ -47,19 +52,21 @@ Description-Content-Type: text/markdown
 Cognitrix is an open-source autonous AI agents orchestrator built in Python. It allows you to create and manage AI agents easily.
 
 ### Features
-- Create and manage AI agents.
-- Conversational interface using chat
-- Modular architecture for easy extensibility
-- Access to various tools like calculators, web - search, etc.
-- Support for multiple platforms
+- **`Agent Creation and Management`:** Create, list, and load AI agents with customizable names, tasks, and configurations.
+- **`LLM Integration`:** Seamlessly integrate with multiple LLM providers, including Anthropic (Claude), Cohere, Groq, Google, OpenAI, and Together.
+- **`Modular Architecture`:** Easily extend the framework by adding new tools, agents, and LLM integrations.
+- **`Conversational Interface`:** Interact with AI agents through a command-line interface, providing queries and receiving responses.
+- **`Tool Integration`:** Agents can utilize a variety of tools, such as calculators, web searches, file system browsers, and more.
+- **`Autonomous Agent Mode`:** Enable agents to operate autonomously, visually perceiving the screen, interacting with UI elements, and performing tasks.
+- **`Multimodal Support`:** Agents can handle both text and image inputs/outputs, enabling multimodal interactions.
 
 ### Architecture
-The core components of Cognitrix are:
+Cognitrix's architecture is designed to be modular and extensible, with core components including:
 
-- `Agent` - Base class for chatbot agents with - support for tools
-- `LLM` - Integration with large language models like - Cohere, GPT-3 etc.
-- `Tools` - Various utility tools like calculators, search etc.
-- `Memory` - For tracking context and state
+1. **`Agents`:** The base Agent class and specialized classes like AIAssistant for creating and managing AI agents.
+2. **`LLMs`:** A collection of classes for integrating with various LLM providers, such as Cohere, OpenAI, Claude, and more.
+3. **`Tools`:** A set of tools that agents can utilize, including calculators, web searches, file system browsers, and more.
+4. **`Templates`:** Customizable prompt templates for guiding the behavior and output formats of LLMs.
 
 The architecture is highly modular and extensible. New tools and capabilities can be easily added.
 
@@ -80,14 +87,14 @@ Run with default settings
 cognitrix
 ```
 
-List supported platforms
+List supported providers
 ```bash
-cognitrix --platforms
+cognitrix --providers
 ```
 
-Run with specific platform
+Run with specific provider
 ```bash
-cognitrix --platform <platform_name>
+cognitrix --provider <provider_name>
 ```
 
 Create a new agent
@@ -116,5 +123,5 @@ Cognitrix is open source and contributions are welcome. See [CONTRIBUTING.md](CO
 ### License
 This project is licensed under the MIT license. See [LICENSE.md](LICENSE.md) for more details.
 
-### Credits
-Cognitrix was created by [Amos Amissah](https://github.com/theonlyamos).
+### Acknowledgments
+Cognitrix was created by [Amos Amissah](https://github.com/theonlyamos) and is heavily inspired by projects like AutoGPT and GPT Engineer. Special thanks to the open-source community for their contributions and the AI companies providing LLM APIs.

cognitrix-0.2.2/README.md

@@ -0,0 +1,77 @@
+### Cognitrix
+Cognitrix is an open-source autonous AI agents orchestrator built in Python. It allows you to create and manage AI agents easily.
+
+### Features
+- **`Agent Creation and Management`:** Create, list, and load AI agents with customizable names, tasks, and configurations.
+- **`LLM Integration`:** Seamlessly integrate with multiple LLM providers, including Anthropic (Claude), Cohere, Groq, Google, OpenAI, and Together.
+- **`Modular Architecture`:** Easily extend the framework by adding new tools, agents, and LLM integrations.
+- **`Conversational Interface`:** Interact with AI agents through a command-line interface, providing queries and receiving responses.
+- **`Tool Integration`:** Agents can utilize a variety of tools, such as calculators, web searches, file system browsers, and more.
+- **`Autonomous Agent Mode`:** Enable agents to operate autonomously, visually perceiving the screen, interacting with UI elements, and performing tasks.
+- **`Multimodal Support`:** Agents can handle both text and image inputs/outputs, enabling multimodal interactions.
+
+### Architecture
+Cognitrix's architecture is designed to be modular and extensible, with core components including:
+
+1. **`Agents`:** The base Agent class and specialized classes like AIAssistant for creating and managing AI agents.
+2. **`LLMs`:** A collection of classes for integrating with various LLM providers, such as Cohere, OpenAI, Claude, and more.
+3. **`Tools`:** A set of tools that agents can utilize, including calculators, web searches, file system browsers, and more.
+4. **`Templates`:** Customizable prompt templates for guiding the behavior and output formats of LLMs.
+
+The architecture is highly modular and extensible. New tools and capabilities can be easily added.
+
+### Getting Started
+
+### Installation
+```bash
+pip install cognitrix
+```
+or
+```bash
+pip install https://github.com/theonlyamos/cognitrix/archive/main.zip
+```
+
+### Usage
+Run with default settings
+```bash
+cognitrix
+```
+
+List supported providers
+```bash
+cognitrix --providers
+```
+
+Run with specific provider
+```bash
+cognitrix --provider <provider_name>
+```
+
+Create a new agent
+```bash
+cognitrix agents --new
+```
+
+List created agents
+```bash
+cognitrix agents
+```
+
+Run with specific agent
+```bash
+cognitrix --agent <agent_name>
+```
+
+Print help message
+```bash
+cognitrix --help
+```
+
+### Contributing
+Cognitrix is open source and contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for more details.
+
+### License
+This project is licensed under the MIT license. See [LICENSE.md](LICENSE.md) for more details.
+
+### Acknowledgments
+Cognitrix was created by [Amos Amissah](https://github.com/theonlyamos) and is heavily inspired by projects like AutoGPT and GPT Engineer. Special thanks to the open-source community for their contributions and the AI companies providing LLM APIs.

{cognitrix-0.2.0 → cognitrix-0.2.2}/cognitrix/__init__.py

@@ -1,3 +1,3 @@
 from .tools import Tool
 from .agents import Agent, AIAssistant
-from .llms import LLM, Clarifai, Cohere, Groq, OpenAI, Together, Gemini
+from .llms import LLM, Clarifai, Cohere, Groq, OpenAI, Together, Google

{cognitrix-0.2.0 → cognitrix-0.2.2}/cognitrix/agents/assistant.py

@@ -1,6 +1,6 @@
 from pydantic import Field
 from ..agents.base import Agent
-from ..agents.templates import ASSISTANT_TEMPLATE
+from ..agents.templates import AUTONOMOUSE_AGENT_2
 
 class AIAssistant(Agent):
     """_summary_
@@ -8,4 +8,4 @@ class AIAssistant(Agent):
     Args:
         Agent (_type_): _description_
     """
-    prompt_template: str = Field(default=ASSISTANT_TEMPLATE)
+    prompt_template: str = Field(default=AUTONOMOUSE_AGENT_2)

cognitrix-0.2.2/cognitrix/agents/base.py

@@ -0,0 +1,417 @@
+import inspect
+import json
+import sys
+import uuid
+import asyncio
+import logging
+import aiofiles
+from pathlib import Path
+from threading import Thread
+from typing import List, Optional, Self, TypeAlias, Union, Type, Any
+from dataclasses import dataclass
+
+from pydantic import BaseModel, Field
+
+from cognitrix.tasks import Task
+from cognitrix.llms.base import LLM
+from cognitrix.tools.base import Tool
+from cognitrix.utils import extract_json, json_return_format
+from cognitrix.agents.templates import AUTONOMOUSE_AGENT_2
+from cognitrix.config import AGENTS_FILE, SESSIONS_FILE
+from cognitrix.llms.session import Session
+from cognitrix.transcriber import Transcriber
+
+logger = logging.getLogger('cognitrix.log')
+
+AgentList: TypeAlias = List['Agent']
+
+class Agent(BaseModel):
+    name: str = Field(default='Agent')
+    llm: LLM
+    tools: List[Tool] = Field(default_factory=list)
+    prompt_template: str = Field(default=AUTONOMOUSE_AGENT_2)
+    verbose: bool = Field(default=False)
+    sub_agents: AgentList = Field(default_factory=list)
+    task: Optional[Task] = None
+    is_sub_agent: bool = Field(default=False)
+    id: str = Field(default_factory=lambda: uuid.uuid4().hex)
+    parent_id: Optional[str] = None
+    autostart: bool = False
+
+    @property
+    def available_tools(self) -> List[str]:
+        return [tool.name for tool in self.tools]
+
+    def format_system_prompt(self):
+        tools_str = self._format_tools_string()
+        subagents_str = self._format_subagents_string()
+        llms_str = self._format_llms_string()
+
+        prompt = self.prompt_template
+        prompt = prompt.replace("{name}", self.name)
+        prompt = prompt.replace("{tools}", tools_str)
+        prompt = prompt.replace("{subagents}", subagents_str)
+        prompt = prompt.replace("{available_tools}", json.dumps(self.available_tools))
+        prompt = prompt.replace("{llms}", llms_str)
+        prompt = prompt.replace("{return_format}", json_return_format)
+
+        if 'json' not in prompt.lower():
+            prompt += f"\n{json_return_format}"
+        
+        self.llm.system_prompt = prompt
+
+    def _format_tools_string(self) -> str:
+        return "You have access to the following Tools:\n" + "\n".join([f"{tool.name}: {tool.description}" for tool in self.tools])
+
+    def _format_subagents_string(self) -> str:
+        subagents_str = "Available Subagents:\n"
+        subagents_str += "\n".join([f"-- {agent.name}: {agent.task.description}" for agent in self.sub_agents if agent.task])
+        subagents_str += "\nYou should always use a subagent for a task if there is one specifically created for that task."
+        subagents_str += "\nWhen creating a sub agent, it's description should be a comprehensive prompt of the agent's behavior, capabilities and example tasks."
+        return subagents_str
+
+    def _format_llms_string(self) -> str:
+        llms = LLM.list_llms()
+        llms_str = "Available LLM Providers:\n" + ", ".join(llms) + "\nChoose one for each subagent."
+        return llms_str
+
+    def generate_prompt(self, query: str | dict, role: str = 'User') -> dict:
+        processed_query = self._process_query(query)
+        prompt: dict[str, Any] = {'role': role, 'type': 'text'}
+
+        if isinstance(processed_query, dict):
+            if self.is_sub_agent:
+                print("=======is sub agent===========")
+                print(processed_query)
+            result = processed_query['result']
+            if isinstance(result, list):
+                if result[0] == 'image':
+                    prompt['type'] = 'image'
+                    prompt['image'] = result[1]
+                elif result[0] == 'agent':
+                    new_agent: Agent = result[1]
+                    new_agent.parent_id = self.id
+                    self.add_sub_agent(new_agent)
+
+                    prompt['message'] = result[2]
+                else:
+                    prompt['message'] = result
+            else:
+                prompt['message'] = result
+        else:
+            prompt['message'] = processed_query
+
+        return prompt
+
+    def _process_query(self, query: str | dict) -> str | dict:
+        return extract_json(query) if isinstance(query, str) else query
+
+    def add_sub_agent(self, agent: 'Agent'):
+        self.sub_agents.append(agent)
+
+    def get_sub_agent_by_name(self, name: str) -> Optional['Agent']:
+        return next((agent for agent in self.sub_agents if agent.name.lower() == name.lower()), None)
+
+    def get_tool_by_name(self, name: str) -> Optional[Tool]:
+        return next((tool for tool in self.tools if tool.name.lower() == name.lower()), None)
+
+    async def process_response(self, response: str) -> Union[dict, str]:
+        # response = response.replace("'", '"')
+        response = response.replace('\\n', '')
+        response = response.replace("'", "\"")
+        # response = response.replace('"', '\\"')
+        response_data = extract_json(response)
+        
+
+        try:
+            if isinstance(response_data, dict):
+                final_result_keys = ['final_answer', 'function_call_result', 'respons']
+
+                if response_data['type'].replace('\\', '') in final_result_keys:
+                    return response_data['result']
+
+                tool = self.get_tool_by_name(response_data['function'])
+                if isinstance(response_data['arguments'], dict):
+                    response_data['arguments'] = list(response_data['arguments'].values())
+
+                if response_data['function'].lower() == 'create agents':
+                    response_data['arguments'] = [*response_data['arguments'], self.id]
+
+                if not tool:
+                    raise Exception(f"Tool {response_data['function']} not found")
+
+                print(f"\nRunning tool '{tool.name.title()}' with parameters: {response_data['arguments']}")
+
+                if 'sub agent' in tool.name.lower():
+                    response_data['arguments'].append(self)
+                    
+                if tool.name.lower() == 'create sub agent':
+                    result = await tool.arun(*response_data['arguments'])
+                else:
+                    result = tool.run(*response_data['arguments'])
+
+                response_json = {
+                    'type': 'function_call_result',
+                    'result': result
+                }
+
+                return response_json
+            else:
+                raise Exception('Not a json object')
+        except Exception as e:
+            # logger.exception(e)
+            return response_data
+
+    def add_tool(self, tool: Tool):
+        self.tools.append(tool)
+
+    def initialize(self, session_id: Optional[str] = None):
+        session: Session = asyncio.run(Session.load(session_id)) if session_id else Session(chat=self.llm.chat_history, agent_id=self.id)
+        self.llm.chat_history = session.chat
+        
+        query: str | dict = input("\nUser (q to quit): ")
+        while True:
+            try:
+                if not query:
+                    query: str | dict = input("\nUser (q to quit): ")
+                    continue
+                if isinstance(query, str):
+                    if query.lower() in ['q', 'quit', 'exit']:
+                        print('Exiting...')
+                        break
+
+                    elif query.lower() == 'add agent':
+                        new_agent = asyncio.run(self.create_agent(is_sub_agent=True, parent_id=self.id))
+                        if new_agent:
+                            self.add_sub_agent(new_agent)
+                            print(f"\nAgent {new_agent.name} added successfully!")
+                        else:
+                            print("\nError creating agent")
+
+                        query = input("\nUser (q to quit): ")
+                        continue
+
+                    elif query.lower() == 'list agents':
+                        agents_str = "\nAvailable Agents:"
+                        sub_agents = [agent for agent in asyncio.run(self.list_agents()) if agent.parent_id == self.id]
+                        for index, agent in enumerate(sub_agents):
+                            agents_str += (f"\n[{index}] {agent.name}")
+                        print(agents_str)
+                        query = input("\nUser (q to quit): ")
+                        continue
+
+                self.format_system_prompt()
+                
+                full_prompt = self.generate_prompt(query)
+                response: Any = self.llm(full_prompt)
+                
+                self.llm.chat_history.append(full_prompt)
+                self.llm.chat_history.append({'role': self.name, 'type': 'text', 'message': response})
+
+                if self.verbose:
+                    print(response)
+
+                result: dict[Any, Any] | str = asyncio.run(self.process_response(response))
+
+                if isinstance(result, dict) and result['type'] == 'function_call_result':
+                    query = result
+                else:
+                    print(f"\n{self.name}: {result}")
+                    query = input("\nUser (q to quit): ")
+                
+                self.save_session(session)
+
+            except KeyboardInterrupt:
+                print('Exiting...')
+                break
+            except Exception as e:
+                logger.exception(e)
+                break
+
+    def start(self, session_id: Optional[str] = None):
+        self.initialize(session_id)
+        
+    def handle_transcription(self, sentence: str, transcriber: Transcriber):
+        if sentence:
+            self.format_system_prompt()
+                
+            full_prompt = self.generate_prompt(sentence)
+            response: Any = self.llm(full_prompt)
+            self.llm.chat_history.append(full_prompt)
+            self.llm.chat_history.append({'role': self.name, 'type': 'text', 'message': response})
+
+            if self.verbose:
+                print(response)
+
+            processsed_response: dict[Any, Any] | str = asyncio.run(self.process_response(response))
+
+            if isinstance(processsed_response, dict) and processsed_response['type'] == 'function_call_result':
+                query = processsed_response
+            else:
+                print(f"\n{self.name}: {processsed_response}")
+                if isinstance(processsed_response, str):
+                    transcriber.text_to_speech(processsed_response)
+
+
+    def start_audio(self, **kwargs):
+        transcriber = Transcriber(on_message_callback=self.handle_transcription)  # Pass callback to Transcriber
+        if transcriber.start_transcription():
+            print("Audio transcription started.")
+            input("\n\nPress Enter to stop recording...\n\n")
+            transcriber.stop_transcription()
+        else:
+            print("Failed to start audio transcription.")
+        
+
+    @staticmethod
+    def start_task_thread(agent: 'Agent', parent: 'Agent'):
+        agent_thread = Thread(target=agent.run_task, args=(parent,))
+        agent_thread.name = agent.name.lower()
+        agent_thread.start()
+
+    def run_task(self, parent: Self):
+        self.format_system_prompt()
+        query = self.task.description if self.task else None
+
+        while query:
+            full_prompt = self.generate_prompt(query)
+            response: Any = self.llm(full_prompt)
+            self.llm.chat_history.append(full_prompt)
+            self.llm.chat_history.append({'role': 'user', 'type': 'text', 'message': response})
+            
+            if parent.verbose:
+                print(response)
+                
+            agent_result = asyncio.run(self.process_response(response))
+            if isinstance(agent_result, dict) and agent_result['type'] == 'function_call_result':
+                query = agent_result
+            else:
+                print(f"\n--{self.name}: {agent_result}")
+                parent_prompt = parent.generate_prompt(response, 'user')
+                parent_prompt['message'] = self.name + ": " + response
+                parent_response: Any = parent.llm(parent_prompt)
+                parent.llm.chat_history.append(parent_prompt)
+                parent.llm.chat_history.append({'role': 'assistant', 'type': 'text', 'message': parent_response})
+                parent_result = asyncio.run(parent.process_response(parent_response))
+                print(f"\n\n{parent.name}: {parent_result}")
+                query = ""
+
+    def call_sub_agent(self, agent_name: str, task_description: str):
+        sub_agent = self.get_sub_agent_by_name(agent_name)
+        if sub_agent:
+            sub_agent.task = Task(description=task_description)
+            if sub_agent.task:
+                self.start_task_thread(sub_agent, self)
+        else:
+            full_prompt = self.generate_prompt(f'Sub-agent with name {agent_name} was not found.')
+            self.llm(full_prompt)
+
+    @classmethod
+    async def create_agent(cls, name: str = '', description: str = '', task_description: str = '', tools: List[Tool] = [],
+                     llm: Optional[LLM] = None, is_sub_agent: bool = False, parent_id=None) -> Optional[Self]:
+        try:
+            name = name or input("\n[Enter agent name]: ")
+
+            while not llm:
+                llms = LLM.list_llms()
+                llms_str = "\nAvailable LLMs:"
+                for index, llm_l in enumerate(llms):
+                    llms_str += (f"\n[{index}] {llm_l}")
+                print(llms_str)
+                agent_llm = int(input("\n[Select LLM]: "))
+                selected_llm = llms[agent_llm]
+                loaded_llm = LLM.load_llm(selected_llm)
+
+                if loaded_llm:
+                    llm = loaded_llm()
+                    if llm:
+                        llm.model = input(f"\nEnter model name [{llm.model}]: ") or llm.model
+                        temp = input(f"\nEnter model temperature [{llm.temperature}]: ")
+                        llm.temperature = float(temp) if temp else llm.temperature
+
+            if llm:
+                task = Task(description=task_description)
+                new_agent = cls(name=name, llm=llm, task=task, tools=tools, is_sub_agent=is_sub_agent, parent_id=parent_id)
+                
+                if description:
+                    new_agent.prompt_template = description
+
+                agents = []
+                async with aiofiles.open(AGENTS_FILE, 'r') as file:
+                    content = await file.read()
+                    agents = json.loads(content) if content else []
+                    agents.append(new_agent.model_dump())
+
+                async with aiofiles.open(AGENTS_FILE, 'w') as file:
+                    await file.write(json.dumps(agents, indent=4))
+
+                return new_agent
+
+        except Exception as e:
+            logger.error(str(e))
+
+    @classmethod
+    async def list_agents(cls, parent_id: Optional[str] = None) -> AgentList:
+        try:
+            agents: AgentList = []
+            async with aiofiles.open(AGENTS_FILE, 'r') as file:
+                content = await file.read()
+                loaded_agents: list[dict] = json.loads(content) if content else []
+                for agent in loaded_agents:
+                    llm = LLM.load_llm(agent["llm"]["provider"])
+                    loaded_agent = Agent(**agent)
+                    if llm:
+                        loaded_agent.llm = llm(**agent["llm"])
+                    agents.append(loaded_agent)
+
+            if parent_id:
+                agents = [agent for agent in agents if agent.parent_id == parent_id]
+
+            return agents
+
+        except Exception as e:
+            logger.exception(e)
+            return []
+
+    @classmethod
+    async def get(cls, id) -> Optional['Agent']:
+        try:
+            agents = await cls.list_agents()
+            loaded_agents: list[Agent] = [agent for agent in agents if agent.id == id]
+            if len(loaded_agents):
+                return loaded_agents[0]
+        except Exception as e:
+            logger.exception(e)
+            return None
+
+    @classmethod
+    def load_agent(cls, agent_name: str) -> Optional['Agent']:
+        try:
+            agent_name = agent_name.lower()
+            agents = asyncio.run(cls.list_agents())
+            loaded_agents: list[Agent] = [agent for agent in agents if agent.name.lower() == agent_name]
+            if len(loaded_agents):
+                agent = loaded_agents[0]
+                agent.sub_agents = asyncio.run(cls.list_agents(agent.id))
+                return agent
+        except Exception as e:
+            logger.exception(e)
+            return None
+    
+    async def save(self):
+        """Save current agent"""
+        agents = await Agent.list_agents()
+        updated_agents = []
+        for index, agent in enumerate(agents):
+            if agent.id == self.id:
+                agents[index] = self
+            
+            updated_agents.append(agent.model_dump())
+        
+        async with aiofiles.open(AGENTS_FILE, 'w') as file:
+            await file.write(json.dumps(updated_agents, indent=4))
+    
+    def save_session(self, session: Session):
+        save_thread = Thread(target=session.save, args=(self.llm.chat_history,))
+        save_thread.daemon = True
+        save_thread.start()