meshagent-computers 0.0.7__tar.gz

meshagent_computers-0.0.7/CHANGELOG.md

@@ -0,0 +1,12 @@
+## [0.0.7]
+- Added code editor agent
+- Fixes
+
+## [0.0.6]
+- Fixes
+
+## [0.0.5]
+- Fix chatbox initialization
+
+## [0.0.4]
+- Streaming support

meshagent_computers-0.0.7/LICENSE

@@ -0,0 +1,201 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction,
+and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by
+the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all
+other entities that control, are controlled by, or are under common
+control with that entity. For the purposes of this definition,
+"control" means (i) the power, direct or indirect, to cause the
+direction or management of such entity, whether by contract or
+otherwise, or (ii) ownership of fifty percent (50%) or more of the
+outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity
+exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications,
+including but not limited to software source code, documentation
+source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical
+transformation or translation of a Source form, including but
+not limited to compiled object code, generated documentation,
+and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or
+Object form, made available under the License, as indicated by a
+copyright notice that is included in or attached to the work
+(an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object
+form, that is based on (or derived from) the Work and for which the
+editorial revisions, annotations, elaborations, or other modifications
+represent, as a whole, an original work of authorship. For the purposes
+of this License, Derivative Works shall not include works that remain
+separable from, or merely link (or bind by name) to the interfaces of,
+the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including
+the original version of the Work and any modifications or additions
+to that Work or Derivative Works thereof, that is intentionally
+submitted to Licensor for inclusion in the Work by the copyright owner
+or by an individual or Legal Entity authorized to submit on behalf of
+the copyright owner. For the purposes of this definition, "submitted"
+means any form of electronic, verbal, or written communication sent
+to the Licensor or its representatives, including but not limited to
+communication on electronic mailing lists, source code control systems,
+and issue tracking systems that are managed by, or on behalf of, the
+Licensor for the purpose of discussing and improving the Work, but
+excluding communication that is conspicuously marked or otherwise
+designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity
+on behalf of whom a Contribution has been received by Licensor and
+subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+copyright license to reproduce, prepare Derivative Works of,
+publicly display, publicly perform, sublicense, and distribute the
+Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+(except as stated in this section) patent license to make, have made,
+use, offer to sell, sell, import, and otherwise transfer the Work,
+where such license applies only to those patent claims licensable
+by such Contributor that are necessarily infringed by their
+Contribution(s) alone or by combination of their Contribution(s)
+with the Work to which such Contribution(s) was submitted. If You
+institute patent litigation against any entity (including a
+cross-claim or counterclaim in a lawsuit) alleging that the Work
+or a Contribution incorporated within the Work constitutes direct
+or contributory patent infringement, then any patent licenses
+granted to You under this License for that Work shall terminate
+as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+Work or Derivative Works thereof in any medium, with or without
+modifications, and in Source or Object form, provided that You
+meet the following conditions:
+
+(a) You must give any other recipients of the Work or
+Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices
+stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works
+that You distribute, all copyright, patent, trademark, and
+attribution notices from the Source form of the Work,
+excluding those notices that do not pertain to any part of
+the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its
+distribution, then any Derivative Works that You distribute must
+include a readable copy of the attribution notices contained
+within such NOTICE file, excluding those notices that do not
+pertain to any part of the Derivative Works, in at least one
+of the following places: within a NOTICE text file distributed
+as part of the Derivative Works; within the Source form or
+documentation, if provided along with the Derivative Works; or,
+within a display generated by the Derivative Works, if and
+wherever such third-party notices normally appear. The contents
+of the NOTICE file are for informational purposes only and
+do not modify the License. You may add Your own attribution
+notices within Derivative Works that You distribute, alongside
+or as an addendum to the NOTICE text from the Work, provided
+that such additional attribution notices cannot be construed
+as modifying the License.
+
+You may add Your own copyright statement to Your modifications and
+may provide additional or different license terms and conditions
+for use, reproduction, or distribution of Your modifications, or
+for any such Derivative Works as a whole, provided Your use,
+reproduction, and distribution of the Work otherwise complies with
+the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+any Contribution intentionally submitted for inclusion in the Work
+by You to the Licensor shall be under the terms and conditions of
+this License, without any additional terms or conditions.
+Notwithstanding the above, nothing herein shall supersede or modify
+the terms of any separate license agreement you may have executed
+with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+names, trademarks, service marks, or product names of the Licensor,
+except as required for reasonable and customary use in describing the
+origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+agreed to in writing, Licensor provides the Work (and each
+Contributor provides its Contributions) on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+implied, including, without limitation, any warranties or conditions
+of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+PARTICULAR PURPOSE. You are solely responsible for determining the
+appropriateness of using or redistributing the Work and assume any
+risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+whether in tort (including negligence), contract, or otherwise,
+unless required by applicable law (such as deliberate and grossly
+negligent acts) or agreed to in writing, shall any Contributor be
+liable to You for damages, including any direct, indirect, special,
+incidental, or consequential damages of any character arising as a
+result of this License or out of the use or inability to use the
+Work (including but not limited to damages for loss of goodwill,
+work stoppage, computer failure or malfunction, or any and all
+other commercial damages or losses), even if such Contributor
+has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+the Work or Derivative Works thereof, You may choose to offer,
+and charge a fee for, acceptance of support, warranty, indemnity,
+or other liability obligations and/or rights consistent with this
+License. However, in accepting such obligations, You may act only
+on Your own behalf and on Your sole responsibility, not on behalf
+of any other Contributor, and only if You agree to indemnify,
+defend, and hold each Contributor harmless for any liability
+incurred by, or claims asserted against, such Contributor by reason
+of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+To apply the Apache License to your work, attach the following
+boilerplate notice, with the fields enclosed by brackets "[]"
+replaced with your own identifying information. (Don't include
+the brackets!)  The text should be enclosed in the appropriate
+comment syntax for the file format. We also recommend that a
+file or class name and description of purpose be included on the
+same "printed page" as the copyright notice for easier
+identification within third-party archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

meshagent_computers-0.0.7/MANIFEST.in

@@ -0,0 +1,2 @@
+include version.py
+include CHANGELOG.md

meshagent_computers-0.0.7/PKG-INFO

@@ -0,0 +1,28 @@
+Metadata-Version: 2.4
+Name: meshagent-computers
+Version: 0.0.7
+Summary: Computer Building Blocks for Meshagent
+Home-page: 
+License: Apache License 2.0
+Project-URL: Documentation, https://meshagent.com
+Project-URL: Website, https://meshagent.com
+Project-URL: Source, https://github.com/meshagent
+Requires-Python: >=3.9.0
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pytest>=8.3.4
+Requires-Dist: pytest-asyncio>=0.24.0
+Requires-Dist: openai>=1.66.2
+Requires-Dist: meshagent-api>=0.0.7
+Requires-Dist: meshagent-agents>=0.0.7
+Requires-Dist: meshagent-tools>=0.0.7
+Requires-Dist: playwright~=1.50.0
+Requires-Dist: browserbase~=1.2.0
+Requires-Dist: scrapybara~=2.4.1
+Dynamic: description-content-type
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary

meshagent_computers-0.0.7/meshagent/computers/__init__.py

@@ -0,0 +1,7 @@
+from .computer import Computer
+from .browserbase import BrowserbaseBrowser
+from .local_playwright import LocalPlaywrightComputer
+from .docker import DockerComputer
+from .scrapybara import ScrapybaraBrowser, ScrapybaraUbuntu
+from .operator import Operator
+from .agent import ComputerAgent

meshagent_computers-0.0.7/meshagent/computers/agent.py

@@ -0,0 +1,225 @@
+from meshagent.openai import OpenAIResponsesAdapter
+from meshagent.agents import LLMAdapter, AgentChatContext
+from meshagent.tools import Tool, Toolkit, ToolContext
+from meshagent.agents.prompt import PromptAgent
+from meshagent.computers import Computer, Operator
+from meshagent.agents.chat import ChatBot, ChatThreadContext
+from meshagent.api import RemoteParticipant, FileResponse
+from meshagent.api.messaging import RawOutputs
+
+from typing import Optional
+import base64
+import json
+import logging
+
+logging.basicConfig()
+logger = logging.getLogger("computer")
+logger.setLevel(logging.INFO)
+
+class ComputerAgent[ComputerType:Computer, OperatorType:Operator](ChatBot):
+    def __init__(self, *, name, 
+            title=None,
+            description=None,
+            requires=None,
+            labels = None,
+            computer_cls: ComputerType,
+            operator_cls: OperatorType,
+            rules: Optional[list[str]] = None,
+            llm_adapter: Optional[LLMAdapter] = None,
+            toolkits: list[Toolkit] = None
+        ):
+
+        if rules == None:
+            rules=[
+                "if asked to go to a URL, you MUST use the goto function to go to the url if it is available",
+                "after going directly to a URL, the screen will change so you should take a look at it to know what to do next"
+            ]
+        super().__init__(
+            name=name,
+            title=title,
+            description=description,
+            requires=requires,
+            labels=labels,
+            llm_adapter=llm_adapter,
+            toolkits=toolkits,
+            rules=rules
+        )
+        self.computer_cls = computer_cls
+        self.operator_cls = operator_cls
+
+
+    async def init_thread_context(self, *, thread_context: ChatThreadContext):
+        
+        operator : Operator = self.operator_cls()
+        computer : Computer = self.computer_cls()
+        started = False
+
+        class ComputerTool(Tool):
+            def __init__(self, *, operator: Operator, computer: Computer, title = "computer_call", description = "handle computer calls from computer use preview", rules = [], thumbnail_url = None, defs = None):
+                super().__init__(
+                    name="computer_call",
+                    # TODO: give a correct schema
+                    input_schema={
+                        "additionalProperties" : False,
+                        "type" : "object",
+                        "required" : [],
+                        "properties" : {}
+                    },
+                    title=title,
+                    description=description,
+                    rules=rules,
+                    thumbnail_url=thumbnail_url,
+                    defs=defs,
+                   
+                )
+                self.computer = computer
+
+
+            @property
+            def options(self):
+                return {
+                    "type": "computer-preview",
+                    "display_width": self.computer.dimensions[0],
+                    "display_height": self.computer.dimensions[1],
+                    "environment": self.computer.environment,
+                }
+
+            async def execute(self,  context: ToolContext, *, arguments):
+                
+                nonlocal started
+                if started == False:
+                    await self.computer.__aenter__()
+                    started = True
+
+                for participant in thread_context.participants:
+                    await context.room.messaging.send_message(
+                        to=participant,
+                        type="computer_use",
+                        message={
+                            "arguments" : arguments
+                        }
+                    )
+
+                outputs = await operator.play(computer=self.computer, item=arguments)
+                for output in outputs:
+                      if output["type"] == "computer_call_output":
+                          if output["output"] != None:
+                              if output["output"]["type"] == "input_image":
+                                  
+                                b64 : str = output["output"]["image_url"]
+                                image_data_b64 = b64.split(",", 1)
+                                
+                                image_bytes = base64.b64decode(image_data_b64[1])
+
+                                for participant in thread_context.participants:
+                                    context.room.messaging.send_message_nowait(
+                                        to=participant,
+                                        type="computer_screen",
+                                        message={
+                                        },
+                                        attachment=image_bytes
+                                    )
+                
+                nonlocal computer_toolkit
+                if len(computer_toolkit.tools) == 1:
+                    # HACK: after looking at the page, add the other tools,
+                    # if we add these first then the computer-use-preview mode fails if it calls them before using the computer
+                    computer_toolkit.tools.extend([
+                        ScreenshotTool(computer=computer),
+                        GotoURL(computer=computer),
+                    ])
+                return RawOutputs(outputs=outputs)
+            
+        class ScreenshotTool(Tool):
+            def __init__(self, computer: Computer):
+                self.computer = computer
+
+                super().__init__(
+                    name="screenshot",
+                    # TODO: give a correct schema
+                    input_schema={
+                        "additionalProperties" : False,
+                        "type" : "object",
+                        "required" : ["full_page","save_path"],
+                        "properties" : {
+                            "full_page" : {
+                                "type" : "boolean"
+                            },
+                            "save_path" : {
+                                "type" : "string",
+                                "description" : "a file path to save the screenshot to (should end with .png)"
+                            }
+                        }
+                    },
+                    description="take a screenshot of the current page",               
+                )
+
+            
+            async def execute(self, context: ToolContext, save_path: str, full_page: bool):
+                nonlocal started
+                if started == False:
+                    await self.computer.__aenter__()
+                    started = True
+
+                screenshot_bytes = await self.computer.screenshot_bytes(full_page=full_page)
+                handle = await context.room.storage.open(path=save_path, overwrite=True)
+                await context.room.storage.write(handle=handle, data=screenshot_bytes)
+                await context.room.storage.close(handle=handle)
+
+                return f"saved screenshot to {save_path}"
+            
+        class GotoURL(Tool):
+            def __init__(self, computer: Computer):
+                self.computer = computer
+
+                super().__init__(
+                    name="goto",
+                    description="goes to a specific URL. Make sure it starts with http:// or https://",
+                    # TODO: give a correct schema
+                    input_schema={
+                        "additionalProperties" : False,
+                        "type" : "object",
+                        "required" : ["url"],
+                        "properties" : {
+                            "url" : {
+                                "type" : "string",
+                                "description": "Fully qualified URL to navigate to.",
+                            }
+                        }
+                    },
+                )
+
+            
+            async def execute(self, context: ToolContext, url: str):
+                nonlocal started
+                if started == False:
+                    await self.computer.__aenter__()
+                    started = True
+
+                if url.startswith("https://") == False and url.startswith("http://") == False:
+                    url = "https://"+url
+
+                await self.computer.goto(url)
+
+                # send an updated screen out
+                for participant in thread_context.participants:
+                    context.room.messaging.send_message_nowait(
+                        to=participant,
+                        type="computer_screen",
+                        message={
+                        },
+                        attachment = await self.computer.screenshot_bytes(full_page=False)
+                    )
+        
+        computer_tool = ComputerTool(computer=computer, operator=operator)
+        
+        computer_toolkit = Toolkit(name="meshagent.openai.computer", tools=[
+            computer_tool
+        ])
+
+        thread_context.toolkits = [
+            computer_toolkit,
+            *thread_context.toolkits
+        ]
+
+

meshagent_computers-0.0.7/meshagent/computers/base_playwright.py

@@ -0,0 +1,179 @@
+import time
+import base64
+from typing import List, Dict, Literal
+from playwright.async_api import async_playwright, Browser, Page, Route, Request
+from meshagent.computers.utils import check_blocklisted_url
+
+# Optional: key mapping if your model uses "CUA" style keys
+CUA_KEY_TO_PLAYWRIGHT_KEY = {
+    "/": "Divide",
+    "\\": "Backslash",
+    "alt": "Alt",
+    "arrowdown": "ArrowDown",
+    "arrowleft": "ArrowLeft",
+    "arrowright": "ArrowRight",
+    "arrowup": "ArrowUp",
+    "backspace": "Backspace",
+    "capslock": "CapsLock",
+    "cmd": "Meta",
+    "ctrl": "Control",
+    "delete": "Delete",
+    "end": "End",
+    "enter": "Enter",
+    "esc": "Escape",
+    "home": "Home",
+    "insert": "Insert",
+    "option": "Alt",
+    "pagedown": "PageDown",
+    "pageup": "PageUp",
+    "shift": "Shift",
+    "space": " ",
+    "super": "Meta",
+    "tab": "Tab",
+    "win": "Meta",
+}
+
+
+class BasePlaywrightComputer:
+    """
+    Abstract base for Playwright-based computers:
+
+      - Subclasses override `_get_browser_and_page()` to do local or remote connection,
+        returning (Browser, Page).
+      - This base class handles context creation (`__enter__`/`__exit__`),
+        plus standard "Computer" actions like click, scroll, etc.
+      - We also have extra browser actions: `goto(url)` and `back()`.
+    """
+
+    environment: Literal["browser"] = "browser"
+    dimensions = (1024, 768)
+
+    def __init__(self):
+        self._playwright = None
+        self._browser: Browser | None = None
+        self._page: Page | None = None
+
+    async def __aenter__(self):
+        # Start Playwright and call the subclass hook for getting browser/page
+        self._context = async_playwright()
+        self._playwright = await self._context.__aenter__()
+        self._browser, self._page = await self._get_browser_and_page()
+
+        # Set up network interception to flag URLs matching domains in BLOCKED_DOMAINS
+        async def handle_route(route: Route, request: Request):
+
+            url = request.url
+            if check_blocklisted_url(url):
+                print(f"Flagging blocked domain: {url}")
+                await route.abort()
+            else:
+                await route.continue_()
+
+        await self._page.route("**/*", handle_route)
+
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        if self._browser:
+            await self._browser.close()
+        if self._playwright:
+            await self._context.__aexit__(exc_type, exc_val, exc_tb)
+
+    def get_current_url(self) -> str:
+        if self._page == None:
+            return "about:blank"
+        
+    async def ensure_page(self):
+        # After a timeout, we might loose our browser
+        if self._page == None or self._browser.is_connected == False:
+            self._browser, self._page = await self._get_browser_and_page()
+
+    # --- Common "Computer" actions ---
+
+    
+    async def screenshot_bytes(self, full_page: bool = False) -> bytes:
+        await self.ensure_page()
+        png_bytes = await self._page.screenshot(full_page=full_page)
+        return png_bytes
+
+    async def screenshot(self, full_page: bool = False) -> str:
+        await self.ensure_page()
+        png_bytes = await self.screenshot_bytes(full_page=full_page)
+        return base64.b64encode(png_bytes).decode("utf-8")
+
+    async def click(self, x: int, y: int, button: str = "left") -> None:
+        await self.ensure_page()
+        match button:
+            case "back":
+                await self.back()
+            case "forward":
+                await self.forward()
+            case "wheel":
+                await self._page.mouse.wheel(x, y)
+            case _:
+                button_mapping = {"left": "left", "right": "right"}
+                button_type = button_mapping.get(button, "left")
+                await self._page.mouse.click(x, y, button=button_type)
+
+    async def double_click(self, x: int, y: int) -> None:
+        await self.ensure_page()
+        await self._page.mouse.dblclick(x, y)
+
+    async def scroll(self, x: int, y: int, scroll_x: int, scroll_y: int) -> None:
+        await self.ensure_page()
+        await self._page.mouse.move(x, y)
+        await self._page.evaluate(f"window.scrollBy({scroll_x}, {scroll_y})")
+
+    async def type(self, text: str) -> None:
+        await self.ensure_page()
+        await self._page.keyboard.type(text)
+
+    async def wait(self, ms: int = 1000) -> None:
+        await self.ensure_page()
+        time.sleep(ms / 1000)
+
+    async def move(self, x: int, y: int) -> None:
+        await self.ensure_page()
+        await self._page.mouse.move(x, y)
+
+    async def keypress(self, keys: List[str]) -> None:
+        await self.ensure_page()
+        for key in keys:
+            mapped_key = CUA_KEY_TO_PLAYWRIGHT_KEY.get(key.lower(), key)
+            await self._page.keyboard.press(mapped_key)
+
+    async def drag(self, path: List[Dict[str, int]]) -> None:
+        await self.ensure_page()
+        if not path:
+            return
+        
+        await self._page.mouse.move(path[0]["x"], path[0]["y"])
+        await self._page.mouse.down()
+        for point in path[1:]:
+            await self._page.mouse.move(point["x"], point["y"])
+        await self._page.mouse.up()
+
+    async def get_current_url(self) -> str:
+        await self.ensure_page()
+        return self._page.url
+
+    # --- Extra browser-oriented actions ---
+    async def goto(self, url: str) -> None:
+        await self.ensure_page()
+        try:
+            return await self._page.goto(url)
+        except Exception as e:
+            print(f"Error navigating to {url}: {e}")
+
+    async def back(self) -> None:
+        await self.ensure_page()
+        return await self._page.go_back()
+
+    async def forward(self) -> None:
+        await self.ensure_page()
+        return await self._page.go_forward()
+
+    # --- Subclass hook ---
+    async def _get_browser_and_page(self) -> tuple[Browser, Page]:
+        """Subclasses must implement, returning (Browser, Page)."""
+        raise NotImplementedError