wcgw 1.3.0__tar.gz → 1.5.0__tar.gz

wcgw-1.5.0/.github/workflows/python-tests.yml

@@ -0,0 +1,30 @@
+name: Python Test
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: "${{ matrix.python-version }}"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+          pip install .[dev]  # Installs dependencies based on pyproject.toml
+      - name: Run tests
+        run: |
+          python -m unittest discover -s tests

{wcgw-1.3.0 → wcgw-1.5.0}/.vscode/settings.json

@@ -9,7 +9,8 @@
     ],
     "mypy-type-checker.args": [
         "--enable-incomplete-feature=NewGenericSyntax",
-        "--strict"
+        "--strict",
+        
     ],
     "editor.formatOnSave": true,
     "editor.renderWhitespace": "selection",

{wcgw-1.3.0 → wcgw-1.5.0}/PKG-INFO

@@ -1,12 +1,13 @@
 Metadata-Version: 2.3
 Name: wcgw
-Version: 1.3.0
+Version: 1.5.0
 Summary: What could go wrong giving full shell access to chatgpt?
 Project-URL: Homepage, https://github.com/rusiaaman/wcgw
 Author-email: Aman Rusia <gapypi@arcfu.com>
-Requires-Python: <3.13,>=3.10
+Requires-Python: <3.13,>=3.11
 Requires-Dist: anthropic>=0.39.0
 Requires-Dist: fastapi>=0.115.0
+Requires-Dist: mcp>=1.0.0
 Requires-Dist: mypy>=1.11.2
 Requires-Dist: nltk>=3.9.1
 Requires-Dist: openai>=1.46.0
@@ -26,82 +27,124 @@ Requires-Dist: uvicorn>=0.31.0
 Requires-Dist: websockets>=13.1
 Description-Content-Type: text/markdown
 
-# Enable shell access on chatgpt.com
-A custom gpt on chatgpt web app to interact with your local shell.
+# Shell and Coding agent on Chatgpt and Claude desktop apps
+
+A custom gpt on chatgpt web/desktop apps to interact with your local shell, edit files, run code, etc.
 
 [![Tests](https://github.com/rusiaaman/wcgw/actions/workflows/python-tests.yml/badge.svg?branch=main)](https://github.com/rusiaaman/wcgw/actions/workflows/python-tests.yml)
 [![Build](https://github.com/rusiaaman/wcgw/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rusiaaman/wcgw/actions/workflows/python-publish.yml)
 
+[New feature] [26-Nov-2024] Claude desktop support for shell, computer-control, coding agent.
+[src/wcgw/client/mcp_server/Readme.md](src/wcgw/client/mcp_server/Readme.md)
+
 ### 🚀 Highlights
+
 - ⚡ **Full Shell Access**: No restrictions, complete control.
 - ⚡ **Create, Execute, Iterate**: Ask the gpt to keep running compiler checks till all errors are fixed, or ask it to keep checking for the status of a long running command till it's done.
-- ⚡ **Interactive Command Handling**: Supports interactive commands using arrow keys, interrupt, and ansi escape sequences. 
+- ⚡ **Interactive Command Handling**: Supports interactive commands using arrow keys, interrupt, and ansi escape sequences.
 - ⚡ **REPL support**: [beta] Supports python/node and other REPL execution.
 
-###  🪜 Steps: 
+## Claude
+Full readme [src/wcgw/client/mcp_server/Readme.md](src/wcgw/client/mcp_server/Readme.md)
+### Setup
+
+Update `claude_desktop_config.json`
+
+```json
+{
+  "mcpServers": {
+    "wcgw": {
+      "command": "uvx",
+      "args": ["--from", "wcgw@latest", "wcgw_mcp"]
+    }
+  }
+}
+```
+
+Then restart claude app.
+You can then ask claude to execute shell commands, read files, edit files, run your code, etc.
+
+## ChatGPT
+
+### 🪜 Steps:
+
 1. Run the [cli client](https://github.com/rusiaaman/wcgw?tab=readme-ov-file#client) in any directory of choice.
 2. Share the generated id with this GPT: `https://chatgpt.com/g/g-Us0AAXkRh-wcgw-giving-shell-access`
 3. The custom GPT can now run any command on your cli
 
+### Client
 
-## Client
 You need to keep running this client for GPT to access your shell. Run it in a version controlled project's root.
 
-### Option 1: using uv [Recommended]
+#### Option 1: using uv [Recommended]
+
 ```sh
 $ curl -LsSf https://astral.sh/uv/install.sh | sh
-$ uv tool run --python 3.12 wcgw@latest
+$ uvx wcgw@latest
 ```
 
-### Option 2: using pip
+#### Option 2: using pip
+
 Supports python >=3.10 and <3.13
+
 ```sh
 $ pip3 install wcgw
 $ wcgw
 ```
 
-
 This will print a UUID that you need to share with the gpt.
 
+### Chat
 
-## Chat
 Open the following link or search the "wcgw" custom gpt using "Explore GPTs" on chatgpt.com
 
 https://chatgpt.com/g/g-Us0AAXkRh-wcgw-giving-shell-access
 
 Finally, let the chatgpt know your user id in any format. E.g., "user_id=<your uuid>" followed by rest of your instructions.
 
-NOTE: you can resume a broken connection 
+NOTE: you can resume a broken connection
 `wcgw --client-uuid $previous_uuid`
 
-# How it works
+### How it works on chatgpt app?
+
 Your commands are relayed through a server to the terminal client. [You could host the server on your own](https://github.com/rusiaaman/wcgw?tab=readme-ov-file#creating-your-own-custom-gpt-and-the-relay-server). For public convenience I've hosted one at https://wcgw.arcfu.com thanks to the gcloud free tier plan.
 
 Chatgpt sends a request to the relay server using the user id that you share with it. The relay server holds a websocket with the terminal client against the user id and acts as a proxy to pass the request.
 
-It's secure in both the directions. Either a malicious actor or a malicious Chatgpt has to correctly guess your UUID for any security breach. 
+It's secure in both the directions. Either a malicious actor or a malicious Chatgpt has to correctly guess your UUID for any security breach.
 
 # Showcase
 
-## Unit tests and github actions
+## Claude desktop
+
+### Resize image and move it to a new dir
+
+![example](https://github.com/rusiaaman/wcgw/blob/main/static/example.jpg?raw=true)
+
+## Chatgpt app
+
+### Unit tests and github actions
+
 [The first version of unit tests and github workflow to test on multiple python versions were written by the custom chatgpt](https://chatgpt.com/share/6717f922-8998-8005-b825-45d4b348b4dd)
 
-## Create a todo app using react + typescript + vite
-![Screenshot](https://github.com/rusiaaman/wcgw/blob/main/static/ss1.png?raw=true)
+### Create a todo app using react + typescript + vite
 
+![Screenshot](https://github.com/rusiaaman/wcgw/blob/main/static/ss1.png?raw=true)
 
 # Privacy
+
 The relay server doesn't store any data. I can't access any information passing through it and only secure channels are used to communicate.
 
 You may host the server on your own and create a custom gpt using the following section.
 
 # Creating your own custom gpt and the relay server.
+
 I've used the following instructions and action json schema to create the custom GPT. (Replace wcgw.arcfu.com with the address to your server)
 
 https://github.com/rusiaaman/wcgw/blob/main/gpt_instructions.txt
 https://github.com/rusiaaman/wcgw/blob/main/gpt_action_json_schema.json
 
-Run the server 
+Run the server
 `gunicorn --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:443 src.wcgw.relay.serve:app  --certfile fullchain.pem  --keyfile  privkey.pem`
 
 If you don't have public ip and domain name, you can use `ngrok` or similar services to get a https address to the api.
@@ -109,16 +152,24 @@ If you don't have public ip and domain name, you can use `ngrok` or similar serv
 The specify the server url in the `wcgw` command like so
 `wcgw --server-url https://your-url/v1/register`
 
-# [Optional] Local shell access with openai API key
+# [Optional] Local shell access with openai API key or anthropic API key
+
+## Openai
 
 Add `OPENAI_API_KEY` and `OPENAI_ORG_ID` env variables.
 
-Clone the repo and run to install `wcgw_local` command
+Then run
+
+`uvx --from wcgw@latest wcgw_local  --limit 0.1` # Cost limit $0.1
+
+You can now directly write messages or press enter key to open vim for multiline message and text pasting.
+
+## Anthropic
 
-`pip install .`
+Add `ANTHROPIC_API_KEY` env variable.
 
-Then run 
+Then run
 
-`wcgw_local  --limit 0.1` # Cost limit $0.1 
+`uvx --from wcgw@latest wcgw_local --claude`
 
 You can now directly write messages or press enter key to open vim for multiline message and text pasting.

{wcgw-1.3.0 → wcgw-1.5.0}/README.md

@@ -1,79 +1,121 @@
-# Enable shell access on chatgpt.com
-A custom gpt on chatgpt web app to interact with your local shell.
+# Shell and Coding agent on Chatgpt and Claude desktop apps
+
+A custom gpt on chatgpt web/desktop apps to interact with your local shell, edit files, run code, etc.
 
 [![Tests](https://github.com/rusiaaman/wcgw/actions/workflows/python-tests.yml/badge.svg?branch=main)](https://github.com/rusiaaman/wcgw/actions/workflows/python-tests.yml)
 [![Build](https://github.com/rusiaaman/wcgw/actions/workflows/python-publish.yml/badge.svg)](https://github.com/rusiaaman/wcgw/actions/workflows/python-publish.yml)
 
+[New feature] [26-Nov-2024] Claude desktop support for shell, computer-control, coding agent.
+[src/wcgw/client/mcp_server/Readme.md](src/wcgw/client/mcp_server/Readme.md)
+
 ### 🚀 Highlights
+
 - ⚡ **Full Shell Access**: No restrictions, complete control.
 - ⚡ **Create, Execute, Iterate**: Ask the gpt to keep running compiler checks till all errors are fixed, or ask it to keep checking for the status of a long running command till it's done.
-- ⚡ **Interactive Command Handling**: Supports interactive commands using arrow keys, interrupt, and ansi escape sequences. 
+- ⚡ **Interactive Command Handling**: Supports interactive commands using arrow keys, interrupt, and ansi escape sequences.
 - ⚡ **REPL support**: [beta] Supports python/node and other REPL execution.
 
-###  🪜 Steps: 
+## Claude
+Full readme [src/wcgw/client/mcp_server/Readme.md](src/wcgw/client/mcp_server/Readme.md)
+### Setup
+
+Update `claude_desktop_config.json`
+
+```json
+{
+  "mcpServers": {
+    "wcgw": {
+      "command": "uvx",
+      "args": ["--from", "wcgw@latest", "wcgw_mcp"]
+    }
+  }
+}
+```
+
+Then restart claude app.
+You can then ask claude to execute shell commands, read files, edit files, run your code, etc.
+
+## ChatGPT
+
+### 🪜 Steps:
+
 1. Run the [cli client](https://github.com/rusiaaman/wcgw?tab=readme-ov-file#client) in any directory of choice.
 2. Share the generated id with this GPT: `https://chatgpt.com/g/g-Us0AAXkRh-wcgw-giving-shell-access`
 3. The custom GPT can now run any command on your cli
 
+### Client
 
-## Client
 You need to keep running this client for GPT to access your shell. Run it in a version controlled project's root.
 
-### Option 1: using uv [Recommended]
+#### Option 1: using uv [Recommended]
+
 ```sh
 $ curl -LsSf https://astral.sh/uv/install.sh | sh
-$ uv tool run --python 3.12 wcgw@latest
+$ uvx wcgw@latest
 ```
 
-### Option 2: using pip
+#### Option 2: using pip
+
 Supports python >=3.10 and <3.13
+
 ```sh
 $ pip3 install wcgw
 $ wcgw
 ```
 
-
 This will print a UUID that you need to share with the gpt.
 
+### Chat
 
-## Chat
 Open the following link or search the "wcgw" custom gpt using "Explore GPTs" on chatgpt.com
 
 https://chatgpt.com/g/g-Us0AAXkRh-wcgw-giving-shell-access
 
 Finally, let the chatgpt know your user id in any format. E.g., "user_id=<your uuid>" followed by rest of your instructions.
 
-NOTE: you can resume a broken connection 
+NOTE: you can resume a broken connection
 `wcgw --client-uuid $previous_uuid`
 
-# How it works
+### How it works on chatgpt app?
+
 Your commands are relayed through a server to the terminal client. [You could host the server on your own](https://github.com/rusiaaman/wcgw?tab=readme-ov-file#creating-your-own-custom-gpt-and-the-relay-server). For public convenience I've hosted one at https://wcgw.arcfu.com thanks to the gcloud free tier plan.
 
 Chatgpt sends a request to the relay server using the user id that you share with it. The relay server holds a websocket with the terminal client against the user id and acts as a proxy to pass the request.
 
-It's secure in both the directions. Either a malicious actor or a malicious Chatgpt has to correctly guess your UUID for any security breach. 
+It's secure in both the directions. Either a malicious actor or a malicious Chatgpt has to correctly guess your UUID for any security breach.
 
 # Showcase
 
-## Unit tests and github actions
+## Claude desktop
+
+### Resize image and move it to a new dir
+
+![example](https://github.com/rusiaaman/wcgw/blob/main/static/example.jpg?raw=true)
+
+## Chatgpt app
+
+### Unit tests and github actions
+
 [The first version of unit tests and github workflow to test on multiple python versions were written by the custom chatgpt](https://chatgpt.com/share/6717f922-8998-8005-b825-45d4b348b4dd)
 
-## Create a todo app using react + typescript + vite
-![Screenshot](https://github.com/rusiaaman/wcgw/blob/main/static/ss1.png?raw=true)
+### Create a todo app using react + typescript + vite
 
+![Screenshot](https://github.com/rusiaaman/wcgw/blob/main/static/ss1.png?raw=true)
 
 # Privacy
+
 The relay server doesn't store any data. I can't access any information passing through it and only secure channels are used to communicate.
 
 You may host the server on your own and create a custom gpt using the following section.
 
 # Creating your own custom gpt and the relay server.
+
 I've used the following instructions and action json schema to create the custom GPT. (Replace wcgw.arcfu.com with the address to your server)
 
 https://github.com/rusiaaman/wcgw/blob/main/gpt_instructions.txt
 https://github.com/rusiaaman/wcgw/blob/main/gpt_action_json_schema.json
 
-Run the server 
+Run the server
 `gunicorn --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:443 src.wcgw.relay.serve:app  --certfile fullchain.pem  --keyfile  privkey.pem`
 
 If you don't have public ip and domain name, you can use `ngrok` or similar services to get a https address to the api.
@@ -81,16 +123,24 @@ If you don't have public ip and domain name, you can use `ngrok` or similar serv
 The specify the server url in the `wcgw` command like so
 `wcgw --server-url https://your-url/v1/register`
 
-# [Optional] Local shell access with openai API key
+# [Optional] Local shell access with openai API key or anthropic API key
+
+## Openai
 
 Add `OPENAI_API_KEY` and `OPENAI_ORG_ID` env variables.
 
-Clone the repo and run to install `wcgw_local` command
+Then run
+
+`uvx --from wcgw@latest wcgw_local  --limit 0.1` # Cost limit $0.1
+
+You can now directly write messages or press enter key to open vim for multiline message and text pasting.
+
+## Anthropic
 
-`pip install .`
+Add `ANTHROPIC_API_KEY` env variable.
 
-Then run 
+Then run
 
-`wcgw_local  --limit 0.1` # Cost limit $0.1 
+`uvx --from wcgw@latest wcgw_local --claude`
 
 You can now directly write messages or press enter key to open vim for multiline message and text pasting.

wcgw-1.5.0/add.py

@@ -0,0 +1,6 @@
+def add_numbers(a: int, b: int) -> int:
+    return a + b
+
+# Test the function
+result = add_numbers(5, 3)
+print(f"5 + 3 = {result}")

wcgw-1.5.0/claude_desktop_config.json

@@ -0,0 +1,15 @@
+{
+  "mcpServers": {
+    "wcgw": {
+      "command": "uvx",
+      "args": [
+        "--from",
+        "wcgw",
+        "wcgw_mcp"
+      ],
+      "protocol": "mcp",
+      "defaultModel": "claude-3-sonnet",
+      "maxTokens": 4096
+    }
+  }
+}

{wcgw-1.3.0 → wcgw-1.5.0}/gpt_instructions.txt

@@ -17,6 +17,7 @@ Instructions for `BashCommand`:
 - Optionally `exit shell has restarted` is the output, in which case environment resets, you can run fresh commands.
 - The first line might be `(...truncated)` if the output is too long.
 - The control will return to you in 5 seconds regardless of the status. For heavy commands, keep checking status using BashInteraction till they are finished.
+- Run long running commands in background using screen instead of "&".
 
 Instructions for `Read File`
 - Read full content of a file.
@@ -24,7 +25,6 @@ Instructions for `Read File`
 
 Instructions for `Create File New`
 - Write content to a new file. Provide file path and content. Use this instead of BashCommand for writing new files.
-- This doesn't create any directories, please create directories using `mkdir -p` BashCommand.
 - Provide absolute file path only.
 - For editing existing files, use FileEdit.
 

{wcgw-1.3.0 → wcgw-1.5.0}/pyproject.toml

@@ -1,10 +1,10 @@
 [project]
 authors = [{ name = "Aman Rusia", email = "gapypi@arcfu.com" }]
 name = "wcgw"
-version = "1.3.0"
+version = "1.5.0"
 description = "What could go wrong giving full shell access to chatgpt?"
 readme = "README.md"
-requires-python = ">=3.10, <3.13"
+requires-python = ">=3.11, <3.13"
 dependencies = [
     "openai>=1.46.0",
     "mypy>=1.11.2",
@@ -25,6 +25,7 @@ dependencies = [
     "semantic-version>=2.10.0",
     "nltk>=3.9.1",
     "anthropic>=0.39.0",
+    "mcp>=1.0.0",
 ]
 
 [project.urls]
@@ -38,6 +39,7 @@ build-backend = "hatchling.build"
 wcgw_local = "wcgw:app"
 wcgw = "wcgw:listen"
 wcgw_relay = "wcgw.relay.serve:run"
+wcgw_mcp = "wcgw:mcp_server"
 
 [tool.uv]
 dev-dependencies = [

{wcgw-1.3.0 → wcgw-1.5.0}/src/wcgw/__init__.py

@@ -1,2 +1,3 @@
 from .client.cli import app
 from .client.tools import run as listen
+from .client.mcp_server import main as mcp_server

wcgw-1.5.0/src/wcgw/client/__main__.py

@@ -0,0 +1,3 @@
+from .cli import app
+
+app()

{wcgw-1.3.0 → wcgw-1.5.0}/src/wcgw/client/anthropic_client.py

@@ -27,15 +27,19 @@ from ..types_ import (
     CreateFileNew,
     FileEditFindReplace,
     FileEdit,
+    Keyboard,
+    Mouse,
     ReadFile,
     ReadImage,
-    Writefile,
     ResetShell,
+    ScreenShot,
+    GetScreenInfo,
 )
 
 from .common import Models, discard_input
 from .common import CostData
 from .tools import ImageData
+from .computer_use import Computer
 
 from .tools import (
     DoneFlag,
@@ -166,6 +170,7 @@ def loop(
 - The first line might be `(...truncated)` if the output is too long.
 - Always run `pwd` if you get any file or directory not found error to make sure you're not lost.
 - The control will return to you in 5 seconds regardless of the status. For heavy commands, keep checking status using BashInteraction till they are finished.
+- Run long running commands in background using screen instead of "&".
 """,
         ),
         ToolParam(
@@ -192,7 +197,6 @@ def loop(
             name="CreateFileNew",
             description="""
 - Write content to a new file. Provide file path and content. Use this instead of BashCommand for writing new files.
-- This doesn't create any directories, please create directories using `mkdir -p` BashCommand.
 - Provide absolute file path only.
 - For editing existing files, use FileEdit instead of this tool.
 """,
@@ -205,7 +209,7 @@ def loop(
         ToolParam(
             input_schema=ResetShell.model_json_schema(),
             name="ResetShell",
-            description="Resets the shell. Use only if all interrupts and prompt reset attempts have failed repeatedly.",
+            description="Resets the shell. Use only if all interrupts and prompt reset attempts have failed repeatedly.\nAlso exits the docker environment.\nYou need to call GetScreenInfo again",
         ),
         ToolParam(
             input_schema=FileEdit.model_json_schema(),
@@ -213,6 +217,46 @@ def loop(
             description="""
 - Use absolute file path only.
 - Use SEARCH/REPLACE blocks to edit the file.
+""",
+        ),
+        ToolParam(
+            input_schema=GetScreenInfo.model_json_schema(),
+            name="GetScreenInfo",
+            description="""
+- Get display information of an OS running on docker using image "ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest"
+- If user hasn't provided docker image id, check using `docker ps` and provide the id.
+- Important: call this first in the conversation before ScreenShot, Mouse, and Keyboard tools.
+- Connects shell to the docker environment.
+- Note: once this is called, the shell enters the docker environment. All bash commands will run over there.
+""",
+        ),
+        ToolParam(
+            input_schema=ScreenShot.model_json_schema(),
+            name="ScreenShot",
+            description="""
+- Capture screenshot of an OS running on docker using image "ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest"
+- If user hasn't provided docker image id, check using `docker ps` and provide the id.
+- Capture ScreenShot of the current screen for automation.
+""",
+        ),
+        ToolParam(
+            input_schema=Mouse.model_json_schema(),
+            name="Mouse",
+            description="""
+- Interact with docker container running image "ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest"
+- If user hasn't provided docker image id, check using `docker ps` and provide the id.
+- Interact with the screen using mouse
+""",
+        ),
+        ToolParam(
+            input_schema=Keyboard.model_json_schema(),
+            name="Keyboard",
+            description="""
+- Interact with docker container running image "ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest"
+- If user hasn't provided docker image id, check using `docker ps` and provide the id.
+- Emulate keyboard input to the screen
+- Uses xdootool to send keyboard input, keys like Return, BackSpace, Escape, Page_Up, etc. can be used.
+- Do not use it to interact with Bash tool.
 """,
         ),
     ]
@@ -358,7 +402,7 @@ System information:
                                 }
                             )
                             try:
-                                output_or_done, _ = get_tool_output(
+                                output_or_dones, _ = get_tool_output(
                                     tool_parsed,
                                     enc,
                                     limit - cost,
@@ -366,45 +410,46 @@ System information:
                                     max_tokens=8000,
                                 )
                             except Exception as e:
-                                output_or_done = (
-                                    f"GOT EXCEPTION while calling tool. Error: {e}"
-                                )
+                                output_or_dones = [
+                                    (f"GOT EXCEPTION while calling tool. Error: {e}")
+                                ]
                                 tb = traceback.format_exc()
-                                error_console.print(output_or_done + "\n" + tb)
-
-                            if isinstance(output_or_done, DoneFlag):
-                                system_console.print(
-                                    f"\n# Task marked done, with output {output_or_done.task_output}",
-                                )
-                                return output_or_done.task_output, cost
-
-                            output = output_or_done
-                            if isinstance(output, ImageData):
-                                tool_results.append(
-                                    ToolResultBlockParam(
-                                        type="tool_result",
-                                        tool_use_id=tc["id"],
-                                        content=[
-                                            {
-                                                "type": "image",
-                                                "source": {
-                                                    "type": "base64",
-                                                    "media_type": output.media_type,
-                                                    "data": output.data,
-                                                },
-                                            }
-                                        ],
+                                error_console.print(str(output_or_dones) + "\n" + tb)
+
+                            if any(isinstance(x, DoneFlag) for x in output_or_dones):
+                                return "", cost
+
+                            tool_results_content: list[
+                                TextBlockParam | ImageBlockParam
+                            ] = []
+                            for output in output_or_dones:
+                                assert not isinstance(output, DoneFlag)
+                                if isinstance(output, ImageData):
+                                    tool_results_content.append(
+                                        {
+                                            "type": "image",
+                                            "source": {
+                                                "type": "base64",
+                                                "media_type": output.media_type,
+                                                "data": output.data,
+                                            },
+                                        }
                                     )
-                                )
 
-                            else:
-                                tool_results.append(
-                                    ToolResultBlockParam(
-                                        type="tool_result",
-                                        tool_use_id=tc["id"],
-                                        content=output,
+                                else:
+                                    tool_results_content.append(
+                                        {
+                                            "type": "text",
+                                            "text": output,
+                                        },
                                     )
+                            tool_results.append(
+                                ToolResultBlockParam(
+                                    type="tool_result",
+                                    tool_use_id=tc["id"],
+                                    content=tool_results_content,
                                 )
+                            )
                         else:
                             _histories.append(
                                 {"role": "assistant", "content": full_response}