octotoolkit 0.1.0__py3-none-any.whl

octotoolkit-0.1.0.dist-info/METADATA

@@ -0,0 +1,427 @@
+Metadata-Version: 2.4
+Name: octotoolkit
+Version: 0.1.0
+Summary: An effective and easy-to-use agentic framework with extendable tools for complex reasoning.
+Author-email: Pan Lu <lupantech@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://octotools.github.io
+Project-URL: Arxiv, https://arxiv.org/pdf/2502.11271
+Project-URL: Demo, https://huggingface.co/spaces/OctoTools/octotools
+Project-URL: Github, https://github.com/octotools/octotools
+Project-URL: Issues, https://github.com/octotools/octotools/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: LICENSE-TASKS
+Requires-Dist: anthropic==0.40.0
+Requires-Dist: openai==1.75.0
+Requires-Dist: together==1.5.4
+Requires-Dist: google-generativeai==0.8.5
+Requires-Dist: diskcache==5.6.3
+Requires-Dist: easyocr==1.7.2
+Requires-Dist: metapub==0.5.12
+Requires-Dist: pillow==11.1.0
+Requires-Dist: platformdirs==4.3.6
+Requires-Dist: pymed==0.8.9
+Requires-Dist: python-dotenv==1.0.1
+Requires-Dist: sympy==1.13.1
+Requires-Dist: tenacity==9.0.0
+Requires-Dist: transformers==4.49.0
+Requires-Dist: wikipedia==1.4.0
+Dynamic: license-file
+
+
+<a name="readme-top"></a>
+
+<div align="center">
+<img src="https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/octotools.svg" alt="OctoTools Logo" width="100">
+</div>
+
+# OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning
+
+<!-- # <span style="color: #d41544">Octo</span><span style="color: #2176bc">Tools</span>: An Agentic Framework with Extendable Tools for Complex Reasoning -->
+
+<!-- <div align="center">
+<h1>
+  <span style="display: inline-block;">
+    <span style="color: #d41544">Octo</span><span style="color: #2176bc">Tools</span>: An Agentic Framework with Extendable Tools for Complex Reasoning
+  </span>
+</h1>
+</div> -->
+
+<!--- BADGES: START --->
+[![GitHub license](https://img.shields.io/badge/License-MIT-green.svg?logo=github)](https://lbesson.mit-license.org/)
+[![Arxiv](https://img.shields.io/badge/arXiv-2502.11271-B31B1B.svg?logo=arxiv)](https://arxiv.org/abs/2502.11271)
+[![Huggingface Demo](https://img.shields.io/badge/Huggingface-Demo-FFD21E.svg?logo=huggingface)](https://huggingface.co/spaces/OctoTools/octotools)
+[![YouTube](https://img.shields.io/badge/YouTube-Tutorial-FF0000?logo=youtube)](https://www.youtube.com/watch?v=4828sGfx7dk)
+[![Tool Cards](https://img.shields.io/badge/Tool_Cards-OctoTools-2176BC?logo=octopusdeploy)](https://octotools.github.io/#tool-cards)
+[![Visualization](https://img.shields.io/badge/Visualization-OctoTools-D41544?logo=octopusdeploy)](https://octotools.github.io/#visualization)
+[![Coverage](https://img.shields.io/badge/Coverage-OctoTools-2176BC.svg?logo=x)](https://x.com/lupantech/status/1892260474320015861)
+[![Discord](https://img.shields.io/badge/Discord-OctoTools-D41544?logo=discord)](https://discord.gg/F4x9m7Cf)
+
+<!--- BADGES: END --->
+
+
+## Updates
+
+
+### News
+
+- **TBD**: We're excited to collaborate with the community to expand OctoTools to more tools, domains, and beyond! Join our [Discord](https://discord.gg/JQjpHDSM) to get started!
+- **2025-04-17** 🚀: Support for a broader range of LLM engines is available now! See the full list of supported LLM engines [here](https://github.com/octotools/octotools?tab=readme-ov-file#supported-llm-engines).
+- **2025-03-08** 📺: Thrilled to have OctoTools featured in a tutorial by [Discover AI](https://www.youtube.com/@code4AI) at YouTube! Watch the engaging video [here](https://www.youtube.com/watch?v=4828sGfx7dk).
+- **2025-02-16** 📄: Our paper is now available as a preprint on ArXiv! Read it [here](https://arxiv.org/abs/2502.11271)!
+
+
+### TODO
+
+Stay tuned, we're working on the following:
+
+- [X] Add support for Anthropic LLM
+- [X] Add support for Together AI LLM
+- [X] Add support for DeepSeek LLM
+- [X] Add support for Gemini LLM
+- [X] Add support for Grok LLM
+- [ ] Add support for vLLM LLM
+- [ ] Release Python package on PyPI (coming soon!)
+
+
+## Get Started
+
+
+### YouTube Tutorial
+
+Excited to have a tutorial video for OctoTools covered by [Discover AI](https://www.youtube.com/@code4AI) at YouTube!
+
+<div align="center">
+  <a href="https://www.youtube.com/watch?v=4828sGfx7dk">
+    <img src="https://img.youtube.com/vi/4828sGfx7dk/maxresdefault.jpg" alt="OctoTools Tutorial" width="100%">
+  </a>
+</div>
+
+
+### Introduction
+
+We introduce **OctoTools**, a training-free, user-friendly, and easily extensible open-source agentic framework designed to tackle complex reasoning across diverse domains. **OctoTools** introduces standardized **tool cards** to encapsulate tool functionality, a **planner** for both high-level and low-level planning, and an **executor** to carry out tool usage. 
+
+**Tool cards** define tool-usage metadata and encapsulate heterogeneous tools, enabling training-free integration of new tools without additional training or framework refinement. (2) The **planner** governs both high-level and low-level planning to address the global objective and refine actions step by step. (3) The **executor** instantiates tool calls by generating executable commands and save structured results in the context. The final answer is summarized from the full trajectory in the context. Furthermore, the *task-specific toolset optimization algorithm* learns a beneficial subset of tools for downstream tasks.
+
+![framework_overall](https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/models/framework_overall.png)
+![framework_example](https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/models/framework_example.png)
+
+We validate **OctoTools**’ generality across 16 diverse tasks (including MathVista, MMLU-Pro, MedQA, and GAIA-Text), achieving substantial average accuracy gains of 9.3% over GPT-4o. Furthermore, **OctoTools** also outperforms AutoGen, GPT-Functions and LangChain by up to 10.6% when given the same set of tools.
+
+<p align="center">  
+    <img src="https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/result/main_scores_bar_chart.png" width="50%">
+    <!-- Text. -->
+</p>
+
+
+### Supported LLM Engines
+
+We support a broad range of LLM engines, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and more.
+
+| Model Family | Engines (Multi-modal) | Engines (Text-Only) | Official Model List |
+|--------------|-------------------|--------------------| -------------------- |
+| OpenAI | `gpt-4-turbo`, `gpt-4o`, `gpt-4o-mini`,  `gpt-4.1`,  `gpt-4.1-mini`, `gpt-4.1-nano`, `o1`, `o3`, `o1-pro`, `o4-mini` | `gpt-3.5-turbo`, `gpt-4`, `o1-mini`, `o3-mini` | [OpenAI Models](https://platform.openai.com/docs/models) |
+| Anthropic | `claude-3-haiku-20240307`, `claude-3-sonnet-20240229`, `claude-3-opus-20240229`, `claude-3-5-sonnet-20240620`, `claude-3-5-sonnet-20241022`, `claude-3-5-haiku-20241022`, `claude-3-7-sonnet-20250219` | | [Anthropic Models](https://docs.anthropic.com/en/docs/about-claude/models/all-models) |
+| TogetherAI | Most multi-modal models, including `meta-llama/Llama-4-Scout-17B-16E-Instruct`, `Qwen/QwQ-32B`, `Qwen/Qwen2-VL-72B-Instruct` | Most text-only models, including `meta-llama/Llama-3-70b-chat-hf`, `Qwen/Qwen2-72B-Instruct` | [TogetherAI Models](https://api.together.ai/models) |
+| DeepSeek |  | `deepseek-chat`, `deepseek-reasoner` | [DeepSeek Models](https://api-docs.deepseek.com/quick_start/pricing) |
+| Gemini | `gemini-1.5-pro`, `gemini-1.5-flash-8b`, `gemini-1.5-flash`, `gemini-2.0-flash-lite`, `gemini-2.0-flash`, `gemini-2.5-pro-preview-03-25` |  |  [Gemini Models](https://ai.google.dev/gemini-api/docs/models) |
+| Grok | `grok-2-vision-1212`, `grok-2-vision`, `grok-2-vision-latest` | `grok-3-mini-fast-beta`, `grok-3-mini-fast`, `grok-3-mini-fast-latest`, `grok-3-mini-beta`, `grok-3-mini`, `grok-3-mini-latest`, `grok-3-fast-beta`, `grok-3-fast`, `grok-3-fast-latest`, `grok-3-beta`, `grok-3`, `grok-3-latest` | [Grok Models](https://docs.x.ai/docs/models#models-and-pricing) |
+<!-- | vllm | Models including  | Models including | [vllm Models](https://docs.vllm.ai/en/latest/models/supported_models.html) | -->
+
+
+## Installation
+
+Currently, there are two ways to install OctoTools. For most use cases, standard installation would suffice. However, to replicate the [benchmarks](https://github.com/octotools/octotools/tree/main/tasks) mentioned in the original paper and to make your own edits to the code, you would need to several bash scripts from Github. An editable installation is recommended.
+
+### 1. Standard Installation
+
+Create a conda environment and install the dependencies:
+
+```sh
+conda create -n octotools python=3.10
+conda activate octotools
+pip install <TODO: FIND A NAME>
+```
+
+Make `.env` file, and set `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `GOOGLE_CX`, etc. For example:
+
+```sh
+# The content of the .env file
+
+# Used for LLM-powered modules and tools
+OPENAI_API_KEY=<your-api-key-here> # If you want to use OpenAI LLM
+ANTHROPIC_API_KEY=<your-api-key-here> # If you want to use Anthropic LLM
+TOGETHER_API_KEY=<your-api-key-here> # If you want to use TogetherAI LLM
+DEEPSEEK_API_KEY=<your-api-key-here> # If you want to use DeepSeek LLM
+GOOGLE_API_KEY=<your-api-key-here> # If you want to use Gemini LLM
+XAI_API_KEY=<your-api-key-here> # If you want to use Grok LLM
+
+# Used for the Google Search tool
+GOOGLE_API_KEY=<your-api-key-here>
+GOOGLE_CX=<your-cx-here>
+
+# Used for the Advanced Object Detector tool (Optional)
+DINO_KEY=<your-dino-key-here>
+```
+
+Obtain a Google API Key and Google CX according to the [Google Custom Search API](https://developers.google.com/custom-search/v1/overview) documation.
+
+
+### 2. Editable Installation 
+
+Start with a fresh new environment:
+```sh
+conda create -n octotools python=3.10
+conda activate octotools
+```
+
+Clone the [github repo](https://github.com/octotools/octotools):
+```sh
+git clone https://github.com/octotools/octotools.git
+```
+
+In the root directory (the directory that contains ``pyproject.toml``), run the following command:
+```sh
+pip install -e .
+```
+
+(Optional) Install `parallel` for running benchmark experiments in parallel:
+
+```sh
+sudo apt-get update
+sudo apt-get install parallel
+```
+
+## Test the Default Solver
+
+In a brand new folder, paste the following code:
+```py
+from octotools.solver import construct_solver
+
+# remember to put your API keys in .env
+import dotenv
+dotenv.load_dotenv()
+
+llm_engine_name = "gpt-4o"
+solver = construct_solver(llm_engine_name=llm_engine_name)
+
+print(solver.solve("What is the capital of France?"))
+# similarly, you could pass in a photo
+# print(solver.solve("What is the name of this item in French?", image_path="<PATH_TO_IMG>"))
+```
+You should be able to see the output at the end, along with all the intermediate content.
+
+A more detailed jupyter notebook tutorial on the pipeline is coming soon. Stay tuned!
+
+
+## Test Tools in the Toolbox (Need Test Scripts from Github)
+
+Using `Python_Code_Generator_Tool` as an example, test the availability of the tool by running the following:
+
+```sh
+cd src/octotools/tools/python_code_generator
+python tool.py
+```
+
+Expected output:
+
+```
+Execution Result: {'printed_output': 'The sum of all the numbers in the list is: 15', 'variables': {'numbers': [1, 2, 3, 4, 5], 'total_sum': 15}}
+```
+
+You can also test all tools available in the toolbox by running the following:
+
+```sh
+cd src/octotools/tools
+source test_all_tools.sh
+```
+
+Expected testing log:
+
+```
+Testing advanced_object_detector...
+✅ advanced_object_detector passed
+
+Testing arxiv_paper_searcher...
+✅ arxiv_paper_searcher passed
+
+...
+
+Testing wikipedia_knowledge_searcher...
+✅ wikipedia_knowledge_searcher passed
+
+Done testing all tools
+Failed: 0
+```
+
+## Run Inference on Benchmarks (Need Bash Scripts from Github)
+
+Using [CLEVR-Math](https://huggingface.co/datasets/dali-does/clevr-math) as an example, run inference on a benchmark by:
+
+```sh
+cd src/octotools/tasks
+
+# Run inference from clevr-math using GPT-4 only
+source clevr-math/run_gpt4o.sh
+
+# Run inference from clevr-math using the base tool
+source clevr-math/run_octotool_base.sh
+
+# Run inference from clevr-math using Octotools with an optimized toolset
+source clevr-math/run_octotools.sh
+```
+
+More benchmarks are available in the [tasks](https://octotools.github.io/#tasks).
+
+
+## Experiments
+
+
+### Main results
+
+To demonstrate the generality of our **OctoTools** framework, we conduct comprehensive evaluations on 16 diverse benchmarks spanning two modalities, five domains, and four reasoning types. These benchmarks encompass a wide range of complex reasoning tasks, including visual understanding, numerical calculation, knowledge retrieval, and multi-step reasoning.
+
+
+<p align="center">
+    <img src="https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/result/result_table_1.png" width="100%">
+    <!-- Text. -->
+</p>
+
+
+More results are available in the [paper](https://arxiv.org/pdf/2502.11271) or at the [project page](https://octotools.github.io/).
+
+
+### In-depth analysis
+
+We provide a set of in-depth analyses to help you understand the framework. For instance, we visualize the tool usage of **OctoTools** and its baselines  from 16 tasks. It turns out that **OctoTools** takes advantage of different external tools to address task-specific challenges. Explore more findings at our [paper](https://arxiv.org/pdf/2502.11271) or the [project page](https://octotools.github.io/#analysis).
+
+<a align="center">
+    <img src="https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/result/tool_usage_ours_baselines.png" width="100%">
+    <!-- Text. -->
+</a>
+
+### Example visualizations
+
+We provide a set of example visualizations to help you understand the framework. Explore them at the [project page](https://octotools.github.io/#visualization).
+
+<p align="center">  
+    <a href="https://octotools.github.io/#visualization">
+        <img src="https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/result/example_visualization.png" width="80%">
+    </a>
+</p>
+
+
+## Customize OctoTools
+
+The design of each tool card is modular relative to the **OctoTools** framework, enabling users to integrate diverse tools without modifying the underlying framework or agent logic. New tool cards can be added, replaced, or updated with minimal effort, making **OctoTools** robust and extensible as tasks grow in complexity.
+
+<p align="center">
+    <a href="https://octotools.github.io/#tool_cards">
+        <img src="https://raw.githubusercontent.com/octotools/octotools/refs/heads/main/assets/models/tool_cards.png" width="100%">
+    </a>
+</p>
+
+To customize **OctoTools** for your own tasks:
+
+1. **Add a new tool card**: Implement your tool following the structure in [existing tools](https://github.com/OctoTools/OctoTools/tree/main/octotools/tools).
+
+2. **Replace or update existing tools**: You can replace or update tools in the toolbox. For example, we provide the [`Object_Detector_Tool`](https://github.com/OctoTools/OctoTools/blob/main/octotools/tools/object_detector/tool.py) to detect objects in images using an open-source model. We also provide an alternative tool called the [`Advanced_Object_Detector_Tool`](https://github.com/OctoTools/OctoTools/blob/main/octotools/tools/advanced_object_detector/tool.py) to detect objects in images using API calls.
+
+3. **Enable tools for your tasks**: You can enable the whole toolset or a subset of tools for your own tasks by setting the `enabled_tools` argument in [tasks/solve.py](https://github.com/OctoTools/OctoTools/blob/main/octotools/tasks/solve.py).
+
+
+## Resources
+
+### Inspiration
+
+This project draws inspiration from several remarkable projects:
+
+- 📕 [Chameleon](https://github.com/lupantech/chameleon-llm) – Chameleon is an early attempt that augments LLMs with tools, which is a major source of inspiration. A journey of a thousand miles begins with a single step.
+- 📘 [TextGrad](https://github.com/mert-y/textgrad) – We admire and appreciate TextGrad for its innovative and elegant framework design.
+- 📗 [AutoGen](https://github.com/microsoft/autogen) – A trending project that excels in building agentic systems.
+- 📙 [LangChain](https://github.com/langchain-ai/langchain) – A powerful framework for constructing agentic systems, known for its rich functionalities.
+
+
+### Citation
+```bibtex
+@article{lu2025octotools,
+    title={OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning},
+    author={Lu, Pan and Chen, Bowen and Liu, Sheng and Thapa, Rahul and Boen, Joseph and Zou, James},
+    journal = {arXiv preprint arXiv:2502.11271},
+    year={2025}
+}
+```
+
+### Our Team
+<table>
+	<tbody>
+		<tr>
+            <td align="center">
+                <a href="https://lupantech.github.io/">
+                    <img src="https://avatars.githubusercontent.com/u/17663606?v=4" width="100;" alt="lupantech"/>
+                    <br />
+                    <sub><b>Pan Lu</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://bowen118.github.io/">
+                    <img src="https://bowen118.github.io/assets/img/prof_pic.jpg" width="100;" alt="bowen118"/>
+                    <br />
+                    <sub><b>Bowen Chen</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://shengliu66.github.io/">
+                    <img src="https://shengliu66.github.io/profile.jpg" width="100;" alt="shengliu66"/>
+                    <br />
+                    <sub><b>Sheng Liu</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://rthapa84.github.io/">
+                    <img src="https://media.licdn.com/dms/image/v2/D5603AQFc9Bdg5VEPxQ/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1683671172066?e=1744848000&v=beta&t=c8FL4wKP0ynJIcB4foStA4G3C4ffkakw9Nm4G8f7-ps" height="100;" alt="rthapa84"/>
+                    <br />
+                    <sub><b>Rahul Thapa</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://dbds.stanford.edu/people/joseph-boen/">
+                    <img src="https://dbds.stanford.edu/wp-content/uploads/2023/08/joseph-boen.jpg)" width="100;" alt="josephboen"/>
+                    <br />
+                    <sub><b>Joseph Boen</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://www.james-zou.com/">
+                    <img src="https://static.wixstatic.com/media/0f3e8f_cfa7e327b97745ddb8c4a66454b5eb3e~mv2.jpg/v1/fill/w_318,h_446,al_c,q_80,usm_0.66_1.00_0.01,enc_avif,quality_auto/46824428A5822_ForWeb.jpg" height="100;" alt="jameszou"/>
+                    <br />
+                    <sub><b>James Zou</b></sub>
+                </a>
+            </td>
+		</tr>
+	<tbody>
+</table>
+
+
+
+### Contributors
+
+We are trully looking forward to the open-source contributions to OctoTools! If you are interested in contributing, collaborating, or reporting issues, don't hesitate to contact us!
+
+We are also looking forward to your feedback and suggestions!
+
+<p align="right" style="font-size: 14px; color: #2176bc; margin-top: 20px;">
+  <a href="#readme-top" style="text-decoration: none; color: blue; font-weight: bold;">
+    ↑ Back to Top ↑
+  </a>
+</p>
+
+
+### Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=octotools/octotools&type=Date)](https://www.star-history.com/#octotools/octotools&Date)

octotoolkit-0.1.0.dist-info/RECORD

@@ -0,0 +1,8 @@
+octotoolkit-0.1.0.dist-info/licenses/LICENSE,sha256=7RNExKr4nElB2rvraqUsdG-ucKuapvDLpWekcQwYmZo,1131
+octotoolkit-0.1.0.dist-info/licenses/LICENSE-TASKS,sha256=fnFw486_iKn2DHuEIUGDI8CTBNoa9NXpD02h3ByKJmE,18650
+octotools/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+octotools/solver.py,sha256=vdEt3dXlXstMn1EjuJGBYNAkiam_n-53HZcEjdjj7nI,8721
+octotoolkit-0.1.0.dist-info/METADATA,sha256=Gb_uovdhsmAuxZGm3L7sFHwlp1OwEtgTCyT_PfmCmtE,19584
+octotoolkit-0.1.0.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
+octotoolkit-0.1.0.dist-info/top_level.txt,sha256=DkbKjE_FtQKc9Nk11yd4QgiC51ddKISGlr6bcSlO7Hw,10
+octotoolkit-0.1.0.dist-info/RECORD,,

octotoolkit-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (78.1.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

octotoolkit-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) 2025 Pan Lu.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

octotoolkit-0.1.0.dist-info/licenses/LICENSE-TASKS

@@ -0,0 +1,395 @@
+Attribution 4.0 International
+
+=======================================================================
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and
+does not provide legal services or legal advice. Distribution of
+Creative Commons public licenses does not create a lawyer-client or
+other relationship. Creative Commons makes its licenses and related
+information available on an "as-is" basis. Creative Commons gives no
+warranties regarding its licenses, any material licensed under their
+terms and conditions, or any related information. Creative Commons
+disclaims all liability for damages resulting from their use to the
+fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and
+conditions that creators and other rights holders may use to share
+original works of authorship and other material subject to copyright
+and certain other rights specified in the public license below. The
+following considerations are for informational purposes only, are not
+exhaustive, and do not form part of our licenses.
+
+     Considerations for licensors: Our public licenses are
+     intended for use by those authorized to give the public
+     permission to use material in ways otherwise restricted by
+     copyright and certain other rights. Our licenses are
+     irrevocable. Licensors should read and understand the terms
+     and conditions of the license they choose before applying it.
+     Licensors should also secure all rights necessary before
+     applying our licenses so that the public can reuse the
+     material as expected. Licensors should clearly mark any
+     material not subject to the license. This includes other CC-
+     licensed material, or material used under an exception or
+     limitation to copyright. More considerations for licensors:
+	wiki.creativecommons.org/Considerations_for_licensors
+
+     Considerations for the public: By using one of our public
+     licenses, a licensor grants the public permission to use the
+     licensed material under specified terms and conditions. If
+     the licensor's permission is not necessary for any reason--for
+     example, because of any applicable exception or limitation to
+     copyright--then that use is not regulated by the license. Our
+     licenses grant only permissions under copyright and certain
+     other rights that a licensor has authority to grant. Use of
+     the licensed material may still be restricted for other
+     reasons, including because others have copyright or other
+     rights in the material. A licensor may make special requests,
+     such as asking that all changes be marked or described.
+     Although not required by our licenses, you are encouraged to
+     respect those requests where reasonable. More_considerations
+     for the public:
+	wiki.creativecommons.org/Considerations_for_licensees
+
+=======================================================================
+
+Creative Commons Attribution 4.0 International Public License
+
+By exercising the Licensed Rights (defined below), You accept and agree
+to be bound by the terms and conditions of this Creative Commons
+Attribution 4.0 International Public License ("Public License"). To the
+extent this Public License may be interpreted as a contract, You are
+granted the Licensed Rights in consideration of Your acceptance of
+these terms and conditions, and the Licensor grants You such rights in
+consideration of benefits the Licensor receives from making the
+Licensed Material available under these terms and conditions.
+
+
+Section 1 -- Definitions.
+
+  a. Adapted Material means material subject to Copyright and Similar
+     Rights that is derived from or based upon the Licensed Material
+     and in which the Licensed Material is translated, altered,
+     arranged, transformed, or otherwise modified in a manner requiring
+     permission under the Copyright and Similar Rights held by the
+     Licensor. For purposes of this Public License, where the Licensed
+     Material is a musical work, performance, or sound recording,
+     Adapted Material is always produced where the Licensed Material is
+     synched in timed relation with a moving image.
+
+  b. Adapter's License means the license You apply to Your Copyright
+     and Similar Rights in Your contributions to Adapted Material in
+     accordance with the terms and conditions of this Public License.
+
+  c. Copyright and Similar Rights means copyright and/or similar rights
+     closely related to copyright including, without limitation,
+     performance, broadcast, sound recording, and Sui Generis Database
+     Rights, without regard to how the rights are labeled or
+     categorized. For purposes of this Public License, the rights
+     specified in Section 2(b)(1)-(2) are not Copyright and Similar
+     Rights.
+
+  d. Effective Technological Measures means those measures that, in the
+     absence of proper authority, may not be circumvented under laws
+     fulfilling obligations under Article 11 of the WIPO Copyright
+     Treaty adopted on December 20, 1996, and/or similar international
+     agreements.
+
+  e. Exceptions and Limitations means fair use, fair dealing, and/or
+     any other exception or limitation to Copyright and Similar Rights
+     that applies to Your use of the Licensed Material.
+
+  f. Licensed Material means the artistic or literary work, database,
+     or other material to which the Licensor applied this Public
+     License.
+
+  g. Licensed Rights means the rights granted to You subject to the
+     terms and conditions of this Public License, which are limited to
+     all Copyright and Similar Rights that apply to Your use of the
+     Licensed Material and that the Licensor has authority to license.
+
+  h. Licensor means the individual(s) or entity(ies) granting rights
+     under this Public License.
+
+  i. Share means to provide material to the public by any means or
+     process that requires permission under the Licensed Rights, such
+     as reproduction, public display, public performance, distribution,
+     dissemination, communication, or importation, and to make material
+     available to the public including in ways that members of the
+     public may access the material from a place and at a time
+     individually chosen by them.
+
+  j. Sui Generis Database Rights means rights other than copyright
+     resulting from Directive 96/9/EC of the European Parliament and of
+     the Council of 11 March 1996 on the legal protection of databases,
+     as amended and/or succeeded, as well as other essentially
+     equivalent rights anywhere in the world.
+
+  k. You means the individual or entity exercising the Licensed Rights
+     under this Public License. Your has a corresponding meaning.
+
+
+Section 2 -- Scope.
+
+  a. License grant.
+
+       1. Subject to the terms and conditions of this Public License,
+          the Licensor hereby grants You a worldwide, royalty-free,
+          non-sublicensable, non-exclusive, irrevocable license to
+          exercise the Licensed Rights in the Licensed Material to:
+
+            a. reproduce and Share the Licensed Material, in whole or
+               in part; and
+
+            b. produce, reproduce, and Share Adapted Material.
+
+       2. Exceptions and Limitations. For the avoidance of doubt, where
+          Exceptions and Limitations apply to Your use, this Public
+          License does not apply, and You do not need to comply with
+          its terms and conditions.
+
+       3. Term. The term of this Public License is specified in Section
+          6(a).
+
+       4. Media and formats; technical modifications allowed. The
+          Licensor authorizes You to exercise the Licensed Rights in
+          all media and formats whether now known or hereafter created,
+          and to make technical modifications necessary to do so. The
+          Licensor waives and/or agrees not to assert any right or
+          authority to forbid You from making technical modifications
+          necessary to exercise the Licensed Rights, including
+          technical modifications necessary to circumvent Effective
+          Technological Measures. For purposes of this Public License,
+          simply making modifications authorized by this Section 2(a)
+          (4) never produces Adapted Material.
+
+       5. Downstream recipients.
+
+            a. Offer from the Licensor -- Licensed Material. Every
+               recipient of the Licensed Material automatically
+               receives an offer from the Licensor to exercise the
+               Licensed Rights under the terms and conditions of this
+               Public License.
+
+            b. No downstream restrictions. You may not offer or impose
+               any additional or different terms or conditions on, or
+               apply any Effective Technological Measures to, the
+               Licensed Material if doing so restricts exercise of the
+               Licensed Rights by any recipient of the Licensed
+               Material.
+
+       6. No endorsement. Nothing in this Public License constitutes or
+          may be construed as permission to assert or imply that You
+          are, or that Your use of the Licensed Material is, connected
+          with, or sponsored, endorsed, or granted official status by,
+          the Licensor or others designated to receive attribution as
+          provided in Section 3(a)(1)(A)(i).
+
+  b. Other rights.
+
+       1. Moral rights, such as the right of integrity, are not
+          licensed under this Public License, nor are publicity,
+          privacy, and/or other similar personality rights; however, to
+          the extent possible, the Licensor waives and/or agrees not to
+          assert any such rights held by the Licensor to the limited
+          extent necessary to allow You to exercise the Licensed
+          Rights, but not otherwise.
+
+       2. Patent and trademark rights are not licensed under this
+          Public License.
+
+       3. To the extent possible, the Licensor waives any right to
+          collect royalties from You for the exercise of the Licensed
+          Rights, whether directly or through a collecting society
+          under any voluntary or waivable statutory or compulsory
+          licensing scheme. In all other cases the Licensor expressly
+          reserves any right to collect such royalties.
+
+
+Section 3 -- License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+  a. Attribution.
+
+       1. If You Share the Licensed Material (including in modified
+          form), You must:
+
+            a. retain the following if it is supplied by the Licensor
+               with the Licensed Material:
+
+                 i. identification of the creator(s) of the Licensed
+                    Material and any others designated to receive
+                    attribution, in any reasonable manner requested by
+                    the Licensor (including by pseudonym if
+                    designated);
+
+                ii. a copyright notice;
+
+               iii. a notice that refers to this Public License;
+
+                iv. a notice that refers to the disclaimer of
+                    warranties;
+
+                 v. a URI or hyperlink to the Licensed Material to the
+                    extent reasonably practicable;
+
+            b. indicate if You modified the Licensed Material and
+               retain an indication of any previous modifications; and
+
+            c. indicate the Licensed Material is licensed under this
+               Public License, and include the text of, or the URI or
+               hyperlink to, this Public License.
+
+       2. You may satisfy the conditions in Section 3(a)(1) in any
+          reasonable manner based on the medium, means, and context in
+          which You Share the Licensed Material. For example, it may be
+          reasonable to satisfy the conditions by providing a URI or
+          hyperlink to a resource that includes the required
+          information.
+
+       3. If requested by the Licensor, You must remove any of the
+          information required by Section 3(a)(1)(A) to the extent
+          reasonably practicable.
+
+       4. If You Share Adapted Material You produce, the Adapter's
+          License You apply must not prevent recipients of the Adapted
+          Material from complying with this Public License.
+
+
+Section 4 -- Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+     to extract, reuse, reproduce, and Share all or a substantial
+     portion of the contents of the database;
+
+  b. if You include all or a substantial portion of the database
+     contents in a database in which You have Sui Generis Database
+     Rights, then the database in which You have Sui Generis Database
+     Rights (but not its individual contents) is Adapted Material; and
+
+  c. You must comply with the conditions in Section 3(a) if You Share
+     all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+
+Section 5 -- Disclaimer of Warranties and Limitation of Liability.
+
+  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
+     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
+     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
+     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
+     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
+     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
+     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
+     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
+     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
+     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
+
+  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
+     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
+     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
+     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
+     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
+     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
+     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
+     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
+     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
+
+  c. The disclaimer of warranties and limitation of liability provided
+     above shall be interpreted in a manner that, to the extent
+     possible, most closely approximates an absolute disclaimer and
+     waiver of all liability.
+
+
+Section 6 -- Term and Termination.
+
+  a. This Public License applies for the term of the Copyright and
+     Similar Rights licensed here. However, if You fail to comply with
+     this Public License, then Your rights under this Public License
+     terminate automatically.
+
+  b. Where Your right to use the Licensed Material has terminated under
+     Section 6(a), it reinstates:
+
+       1. automatically as of the date the violation is cured, provided
+          it is cured within 30 days of Your discovery of the
+          violation; or
+
+       2. upon express reinstatement by the Licensor.
+
+     For the avoidance of doubt, this Section 6(b) does not affect any
+     right the Licensor may have to seek remedies for Your violations
+     of this Public License.
+
+  c. For the avoidance of doubt, the Licensor may also offer the
+     Licensed Material under separate terms or conditions or stop
+     distributing the Licensed Material at any time; however, doing so
+     will not terminate this Public License.
+
+  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+     License.
+
+
+Section 7 -- Other Terms and Conditions.
+
+  a. The Licensor shall not be bound by any additional or different
+     terms or conditions communicated by You unless expressly agreed.
+
+  b. Any arrangements, understandings, or agreements regarding the
+     Licensed Material not stated herein are separate from and
+     independent of the terms and conditions of this Public License.
+
+
+Section 8 -- Interpretation.
+
+  a. For the avoidance of doubt, this Public License does not, and
+     shall not be interpreted to, reduce, limit, restrict, or impose
+     conditions on any use of the Licensed Material that could lawfully
+     be made without permission under this Public License.
+
+  b. To the extent possible, if any provision of this Public License is
+     deemed unenforceable, it shall be automatically reformed to the
+     minimum extent necessary to make it enforceable. If the provision
+     cannot be reformed, it shall be severed from this Public License
+     without affecting the enforceability of the remaining terms and
+     conditions.
+
+  c. No term or condition of this Public License will be waived and no
+     failure to comply consented to unless expressly agreed to by the
+     Licensor.
+
+  d. Nothing in this Public License constitutes or may be interpreted
+     as a limitation upon, or waiver of, any privileges and immunities
+     that apply to the Licensor or You, including from the legal
+     processes of any jurisdiction or authority.
+
+
+=======================================================================
+
+Creative Commons is not a party to its public
+licenses. Notwithstanding, Creative Commons may elect to apply one of
+its public licenses to material it publishes and in those instances
+will be considered the “Licensor.” The text of the Creative Commons
+public licenses is dedicated to the public domain under the CC0 Public
+Domain Dedication. Except for the limited purpose of indicating that
+material is shared under a Creative Commons public license or as
+otherwise permitted by the Creative Commons policies published at
+creativecommons.org/policies, Creative Commons does not authorize the
+use of the trademark "Creative Commons" or any other trademark or logo
+of Creative Commons without its prior written consent including,
+without limitation, in connection with any unauthorized modifications
+to any of its public licenses or any other arrangements,
+understandings, or agreements concerning use of licensed material. For
+the avoidance of doubt, this paragraph does not form part of the
+public licenses.
+
+Creative Commons may be contacted at creativecommons.org.

octotoolkit-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+octotools

octotools/solver.py

@@ -0,0 +1,231 @@
+import os
+import sys
+import json
+import argparse
+import time
+from typing import Optional
+
+from octotools.models.initializer import Initializer
+from octotools.models.planner import Planner
+from octotools.models.memory import Memory
+from octotools.models.executor import Executor
+from octotools.models.utils import make_json_serializable_truncated
+
+class Solver:
+    def __init__(
+        self,
+        planner,
+        memory,
+        executor,
+        output_types: str = "base,final,direct",
+        max_steps: int = 10,
+        max_time: int = 300,
+        max_tokens: int = 4000,
+        root_cache_dir: str = "cache"
+    ):
+        self.planner = planner
+        self.memory = memory
+        self.executor = executor
+        self.max_steps = max_steps
+        self.max_time = max_time
+        self.max_tokens = max_tokens
+        self.root_cache_dir = root_cache_dir
+
+        self.output_types = output_types.lower().split(',')
+        assert all(output_type in ["base", "final", "direct"] for output_type in self.output_types), "Invalid output type. Supported types are 'base', 'final', 'direct'."
+
+    def solve(self, question: str, image_path: Optional[str] = None):
+        """
+        Solve a single problem from the benchmark dataset.
+        
+        Args:
+            index (int): Index of the problem to solve
+        """
+        # Update cache directory for the executor
+        self.executor.set_query_cache_dir(self.root_cache_dir)
+
+        # Initialize json_data with basic problem information
+        json_data = {
+            "query": question,
+            "image": image_path
+        }
+
+        # Generate base response if requested
+        if 'base' in self.output_types:
+            base_response = self.planner.generate_base_response(question, image_path, self.max_tokens)
+            json_data["base_response"] = base_response
+
+        # If only base response is needed, save and return
+        if set(self.output_types) == {'base'}:
+            return json_data
+    
+        # Continue with query analysis and tool execution if final or direct responses are needed
+        if {'final', 'direct'} & set(self.output_types):
+
+             # Analyze query
+            query_analysis = self.planner.analyze_query(question, image_path)
+            json_data["query_analysis"] = query_analysis
+
+            start_time = time.time()
+            step_count = 0
+            action_times = []
+
+            # Main execution loop
+            while step_count < self.max_steps and (time.time() - start_time) < self.max_time:
+                step_count += 1
+
+                # Generate next step
+                start_time_step = time.time()
+                next_step = self.planner.generate_next_step(
+                    question, 
+                    image_path, 
+                    query_analysis, 
+                    self.memory, 
+                    step_count, 
+                    self.max_steps
+                )
+                context, sub_goal, tool_name = self.planner.extract_context_subgoal_and_tool(next_step)
+
+
+                if tool_name is None or tool_name not in self.planner.available_tools:
+                    print(f"Error: Tool '{tool_name}' is not available or not found.")
+                    command = "Not command is generated due to the tool not found."
+                    result = "Not result is generated due to the tool not found."
+
+                else:
+                    # Generate the tool command
+                    tool_command = self.executor.generate_tool_command(
+                        question, 
+                        image_path, 
+                        context, 
+                        sub_goal, 
+                        tool_name, 
+                        self.planner.toolbox_metadata[tool_name]
+                    )
+                    explanation, command = self.executor.extract_explanation_and_command(tool_command)
+                    
+
+                    # Execute the tool command
+                    result = self.executor.execute_tool_command(tool_name, command)
+                    print("!!! type of result: ", type(result))
+
+                    result = make_json_serializable_truncated(result) # Convert to JSON serializable format
+
+
+                # Track execution time
+                end_time_step = time.time()
+                execution_time_step = round(end_time_step - start_time_step, 2)
+                action_times.append(execution_time_step)
+
+                # Update memory
+                self.memory.add_action(step_count, tool_name, sub_goal, command, result)
+                memeory_actions = self.memory.get_actions()
+
+                # Verify memory
+                stop_verification = self.planner.verificate_context(
+                    question, 
+                    image_path, 
+                    query_analysis, 
+                    self.memory
+                )
+                conclusion = self.planner.extract_conclusion(stop_verification)
+                
+
+                if conclusion == 'STOP':
+                    break
+
+
+            # Add memory and statistics to json_data
+            json_data.update({
+                "memory": memeory_actions,
+                "step_count": step_count,
+                "execution_time": round(time.time() - start_time, 2),
+            })
+
+            # Generate final output if requested
+            if 'final' in self.output_types:
+                final_output = self.planner.generate_final_output(question, image_path, self.memory)
+                json_data["final_output"] = final_output
+
+            # Generate direct output if requested
+            if 'direct' in self.output_types:
+                direct_output = self.planner.generate_direct_output(question, image_path, self.memory)
+                json_data["direct_output"] = direct_output
+
+        return json_data
+            
+def parse_arguments():
+    parser = argparse.ArgumentParser(description="Run the octotools demo with specified parameters.")
+    parser.add_argument("--llm_engine_name", default="gpt-4o", help="LLM engine name.")
+    parser.add_argument(
+        "--output_types",
+        default="base,final,direct",
+        help="Comma-separated list of required outputs (base,final,direct)"
+    )
+    parser.add_argument("--enabled_tools", default="Generalist_Solution_Generator_Tool", help="List of enabled tools.")
+    parser.add_argument("--root_cache_dir", default="solver_cache", help="Path to solver cache directory.")
+    parser.add_argument("--max_tokens", type=int, default=4000, help="Maximum tokens for LLM generation.")
+    parser.add_argument("--max_steps", type=int, default=10, help="Maximum number of steps to execute.")
+    parser.add_argument("--max_time", type=int, default=300, help="Maximum time allowed in seconds.")
+    parser.add_argument("--verbose", type=bool, default=True, help="Enable verbose output.")
+    return parser.parse_args()
+
+def construct_solver(llm_engine_name : str = "gpt-4o",
+                     enabled_tools : list[str] = ["all"],
+                     output_types : str = "direct",
+                     max_steps : int = 10,
+                     max_time : int = 300,
+                     max_tokens : int = 4000,
+                     root_cache_dir : str = "solver_cache"
+                     ):
+    # Instantiate Initializer
+    initializer = Initializer(
+        enabled_tools=enabled_tools,
+        model_string=llm_engine_name
+    )
+
+    # Instantiate Planner
+    planner = Planner(
+        llm_engine_name=llm_engine_name,
+        toolbox_metadata=initializer.toolbox_metadata,
+        available_tools=initializer.available_tools,
+    )
+
+    # Instantiate Memory
+    memory = Memory()
+
+    # Instantiate Executor
+    executor = Executor(
+        llm_engine_name=llm_engine_name,
+        root_cache_dir=root_cache_dir
+    )
+
+
+    # Instantiate Solver
+    solver = Solver(
+        planner=planner,
+        memory=memory,
+        executor=executor,
+        output_types=output_types,
+        max_steps=max_steps,
+        max_time=max_time,
+        max_tokens=max_tokens,
+        root_cache_dir=root_cache_dir
+    )
+
+    return solver
+
+def main(args):
+    solver = construct_solver(llm_engine_name=args.llm_engine_name, 
+                              enabled_tools=args.enabled_tools, 
+                              output_types=args.output_types, 
+                              max_steps=args.max_steps, 
+                              max_time=args.max_time, 
+                              max_tokens=args.max_tokens, 
+                              root_cache_dir=args.root_cache_dir)
+    # Solve the task or problem
+    solver.solve("What is the capital of France?")
+
+if __name__ == "__main__":
+    args = parse_arguments()
+    main(args)