byllm 0.4.1__tar.gz

byllm-0.4.1/PKG-INFO

@@ -0,0 +1,102 @@
+Metadata-Version: 2.3
+Name: byllm
+Version: 0.4.1
+Summary: byLLM Provides Easy to use APIs for different LLM Providers to be used with Jaseci's Jaclang Programming Language.
+License: MIT
+Keywords: llm,jaclang,jaseci,byLLM
+Author: Jason Mars
+Author-email: jason@jaseci.org
+Maintainer: Jason Mars
+Maintainer-email: jason@jaseci.org
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 2
+Classifier: Programming Language :: Python :: 2.7
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.4
+Classifier: Programming Language :: Python :: 3.5
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Provides-Extra: tools
+Provides-Extra: video
+Requires-Dist: jaclang (==0.8.5)
+Requires-Dist: litellm (>=1.75.5.post1)
+Requires-Dist: loguru (>=0.7.2,<0.8.0)
+Requires-Dist: pillow (>=10.4.0,<10.5.0)
+Description-Content-Type: text/markdown
+
+# byLLM - AI Integration Framework for Jac-lang
+
+[![PyPI version](https://img.shields.io/pypi/v/mtllm.svg)](https://pypi.org/project/mtllm/) [![tests](https://github.com/jaseci-labs/jaseci/actions/workflows/test-jaseci.yml/badge.svg?branch=main)](https://github.com/jaseci-labs/jaseci/actions/workflows/test-jaseci.yml)
+
+Meaning Typed Programming (MTP) is a programming paradigm for AI integration where prompt engineering is hidden through code semantics. byLLM is the plugin built, exploring this hypothesis. byLLM is built as a plugin to the Jaseci ecosystem. This plugin can be installed as a PyPI package.
+
+```bash
+pip install byllm
+```
+
+## Basic Example
+
+A basic usecase of MTP can be demonstrated as follows:
+
+```python
+import from byllm {Model}
+
+glob llm = Model(model_name="openai\gpt-4o");
+
+def translate_to(language: str, phrase: str) -> str by llm();
+
+with entry {
+    output = translate_to(language="Welsh", phrase="Hello world");
+    print(output);
+}
+```
+
+## AI-Powered Object Generation
+
+```python
+import from byllm {Model}
+
+glob llm = Model(model_name="gpt-4o");
+
+obj Task {
+    has description: str,
+        priority: int,
+        estimated_time: int;
+}
+
+sem Task.priority = "priority between 0 (highest priority) and 10(lowest priority)";
+
+def create_task(description: str, previous_tasks: list[Task]) -> Task by llm();
+
+with entry {
+    tasks = [];
+    new_task = create_task("Write documentation for the API", tasks);
+    print(f"Task: {new_task.description}, Priority: {new_task.priority}, Time: {new_task.estimated_time}min");
+}
+```
+
+The `by` abstraction allows to automate semantic extraction from existing code semantics, eliminating manual prompt engineering while leveraging type annotations for structured AI responses.
+
+## Documentation and Examples
+
+**📚 Full Documentation**: [Jac byLLM Documentation](https://www.jac-lang.org/learn/jac-byllm/with_llm/)
+
+**🎮 Complete Examples**:
+- [Fantasy Trading Game](https://www.jac-lang.org/learn/examples/mtp_examples/fantasy_trading_game/) - Interactive RPG with AI-generated characters
+- [RPG Level Generator](https://www.jac-lang.org/learn/examples/mtp_examples/rpg_game/) - AI-powered game level creation
+- [RAG Chatbot Tutorial](https://www.jac-lang.org/learn/examples/rag_chatbot/Overview/) - Building chatbots with document retrieval
+
+**🔬 Research**: The research journey of MTP is available on [Arxiv](https://arxiv.org/abs/2405.08965).
+
+## Quick Links
+
+- [Getting Started Guide](https://www.jac-lang.org/learn/jac-byllm/with_llm/)
+- [Model Configuration](https://www.jac-lang.org/learn/jac-byllm/model_declaration/)
+- [Jac Language Documentation](https://www.jac-lang.org/)
+- [GitHub Repository](https://github.com/jaseci-labs/jaseci)

byllm-0.4.1/README.md

@@ -0,0 +1,70 @@
+# byLLM - AI Integration Framework for Jac-lang
+
+[![PyPI version](https://img.shields.io/pypi/v/mtllm.svg)](https://pypi.org/project/mtllm/) [![tests](https://github.com/jaseci-labs/jaseci/actions/workflows/test-jaseci.yml/badge.svg?branch=main)](https://github.com/jaseci-labs/jaseci/actions/workflows/test-jaseci.yml)
+
+Meaning Typed Programming (MTP) is a programming paradigm for AI integration where prompt engineering is hidden through code semantics. byLLM is the plugin built, exploring this hypothesis. byLLM is built as a plugin to the Jaseci ecosystem. This plugin can be installed as a PyPI package.
+
+```bash
+pip install byllm
+```
+
+## Basic Example
+
+A basic usecase of MTP can be demonstrated as follows:
+
+```python
+import from byllm {Model}
+
+glob llm = Model(model_name="openai\gpt-4o");
+
+def translate_to(language: str, phrase: str) -> str by llm();
+
+with entry {
+    output = translate_to(language="Welsh", phrase="Hello world");
+    print(output);
+}
+```
+
+## AI-Powered Object Generation
+
+```python
+import from byllm {Model}
+
+glob llm = Model(model_name="gpt-4o");
+
+obj Task {
+    has description: str,
+        priority: int,
+        estimated_time: int;
+}
+
+sem Task.priority = "priority between 0 (highest priority) and 10(lowest priority)";
+
+def create_task(description: str, previous_tasks: list[Task]) -> Task by llm();
+
+with entry {
+    tasks = [];
+    new_task = create_task("Write documentation for the API", tasks);
+    print(f"Task: {new_task.description}, Priority: {new_task.priority}, Time: {new_task.estimated_time}min");
+}
+```
+
+The `by` abstraction allows to automate semantic extraction from existing code semantics, eliminating manual prompt engineering while leveraging type annotations for structured AI responses.
+
+## Documentation and Examples
+
+**📚 Full Documentation**: [Jac byLLM Documentation](https://www.jac-lang.org/learn/jac-byllm/with_llm/)
+
+**🎮 Complete Examples**:
+- [Fantasy Trading Game](https://www.jac-lang.org/learn/examples/mtp_examples/fantasy_trading_game/) - Interactive RPG with AI-generated characters
+- [RPG Level Generator](https://www.jac-lang.org/learn/examples/mtp_examples/rpg_game/) - AI-powered game level creation
+- [RAG Chatbot Tutorial](https://www.jac-lang.org/learn/examples/rag_chatbot/Overview/) - Building chatbots with document retrieval
+
+**🔬 Research**: The research journey of MTP is available on [Arxiv](https://arxiv.org/abs/2405.08965).
+
+## Quick Links
+
+- [Getting Started Guide](https://www.jac-lang.org/learn/jac-byllm/with_llm/)
+- [Model Configuration](https://www.jac-lang.org/learn/jac-byllm/model_declaration/)
+- [Jac Language Documentation](https://www.jac-lang.org/)
+- [GitHub Repository](https://github.com/jaseci-labs/jaseci)

byllm-0.4.1/byllm/__init__.py

@@ -0,0 +1,8 @@
+"""byLLM Package."""
+
+from byllm.llm import Model
+from byllm.mtir import MTIR
+from byllm.plugin import by
+from byllm.types import Image, MockToolCall, Video
+
+__all__ = ["by", "Image", "MockToolCall", "Model", "MTIR", "Video"]

byllm-0.4.1/byllm/llm.py

@@ -0,0 +1,101 @@
+"""LLM abstraction module.
+
+This module provides a LLM class that abstracts LiteLLM and offers
+enhanced functionality and interface for language model operations.
+"""
+
+# flake8: noqa: E402
+
+import os
+from typing import Generator
+
+from byllm.mtir import MTIR
+
+# This will prevent LiteLLM from fetching pricing information from
+# the bellow URL every time we import the litellm and use a cached
+# local json file. Maybe we we should conditionally enable this.
+# https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+from .llm_connector import LLMConnector
+from .types import CompletionResult
+
+SYSTEM_PERSONA = """\
+This is a task you must complete by returning only the output.
+Do not include explanations, code, or extra text—only the result.
+"""  # noqa E501
+
+INSTRUCTION_TOOL = """
+Use the tools provided to reach the goal. Call one tool at a time with \
+proper args—no explanations, no narration. Think step by step, invoking tools \
+as needed. When done, always call finish_tool(output) to return the final \
+output. Only use tools.
+"""  # noqa E501
+
+
+class Model:
+    """A wrapper class that abstracts LiteLLM functionality.
+
+    This class provides a simplified and enhanced interface for interacting
+    with various language models through LiteLLM.
+    """
+
+    def __init__(self, model_name: str, **kwargs: object) -> None:
+        """Initialize the JacLLM instance.
+
+        Args:
+            model: The model name to use (e.g., "gpt-3.5-turbo", "claude-3-sonnet-20240229")
+            api_key: API key for the model provider
+            **kwargs: Additional configuration options
+        """
+        self.llm_connector = LLMConnector.for_model(model_name, **kwargs)
+
+    def __call__(self, **kwargs: object) -> "Model":
+        """Construct the call parameters and return self (factory pattern).
+
+        Example:
+            ```jaclang
+            llm = JacLLM(model="gpt-3.5-turbo", api_key="your_api_key")
+
+            # The bellow call will construct the parameter and return self.
+            def answer_user_query(query: str) -> str by
+                llm(
+                    temperature=0.7,
+                    max_tokens=100,
+                );
+            ```
+        """
+        self.llm_connector.call_params = kwargs
+        return self
+
+    @property
+    def call_params(self) -> dict[str, object]:
+        """Get the call parameters for the LLM."""
+        return self.llm_connector.call_params
+
+    def invoke(self, mtir: MTIR) -> object:
+        """Invoke the LLM with the given caller and arguments."""
+        if mtir.stream:
+            return self._completion_streaming(mtir)
+
+        # Invoke the LLM and handle tool calls.
+        while True:
+            resp = self._completion_no_streaming(mtir)
+            if resp.tool_calls:
+                for tool_call in resp.tool_calls:
+                    if tool_call.is_finish_call():
+                        return tool_call.get_output()
+                    else:
+                        mtir.add_message(tool_call())
+            else:
+                break
+
+        return resp.output
+
+    def _completion_no_streaming(self, mtir: MTIR) -> CompletionResult:
+        """Perform a completion request with the LLM."""
+        return self.llm_connector.dispatch_no_streaming(mtir)
+
+    def _completion_streaming(self, mtir: MTIR) -> Generator[str, None, None]:
+        """Perform a streaming completion request with the LLM."""
+        return self.llm_connector.dispatch_streaming(mtir)

byllm-0.4.1/byllm/llm_connector.py

@@ -0,0 +1,228 @@
+"""LLM Connector for Litellm, MockLLM, Proxy server, etc.
+
+This module provides an abstract base class for LLM connectors and concrete implementations
+for different LLM services. It includes methods for dispatching requests and handling responses.
+"""
+
+# flake8: noqa: E402
+
+import json
+import logging
+import os
+import random
+import time
+from abc import ABC, abstractmethod
+from typing import Generator, override
+
+# This will prevent LiteLLM from fetching pricing information from
+# the bellow URL every time we import the litellm and use a cached
+# local json file. Maybe we we should conditionally enable this.
+# https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+import litellm
+from litellm._logging import _disable_debugging
+
+from openai import OpenAI
+
+from .mtir import MTIR
+
+from .types import (
+    CompletionResult,
+    LiteLLMMessage,
+    MockToolCall,
+    ToolCall,
+)
+
+DEFAULT_BASE_URL = "http://localhost:4000"
+MODEL_MOCK = "mockllm"
+
+
+class LLMConnector(ABC):
+    """Abstract base class for LLM connectors."""
+
+    def __init__(self, model_name: str, **kwargs: object) -> None:
+        """Initialize the LLM connector with a model."""
+        self.model_name = model_name
+        self.config = kwargs
+        # The parameters for the llm call like temprature, top_k, max_token, etc.
+        # This is only applicable for the next call passed from `by llm(**kwargs)`.
+        self.call_params: dict[str, object] = {}
+
+    @staticmethod
+    def for_model(model_name: str, **kwargs: object) -> "LLMConnector":
+        """Construct the appropriate LLM connector based on the model name."""
+        if model_name.lower().strip() == MODEL_MOCK:
+            return MockLLMConnector(model_name, **kwargs)
+        if kwargs.get("proxy_url"):
+            kwargs["base_url"] = kwargs.pop("proxy_url")
+            return LiteLLMConnector(True, model_name, **kwargs)
+        return LiteLLMConnector(False, model_name, **kwargs)
+
+    def make_model_params(self, mtir: MTIR) -> dict:
+        """Prepare the parameters for the LLM call."""
+        params = {
+            "model": self.model_name,
+            "api_base": (
+                self.config.get("base_url")
+                or self.config.get("host")
+                or self.config.get("api_base")
+            ),
+            "api_key": self.config.get("api_key"),
+            "messages": mtir.get_msg_list(),
+            "tools": mtir.get_tool_list() or None,
+            "response_format": mtir.get_output_schema(),
+            "temperature": self.call_params.get("temperature", 0.7),
+            "max_tokens": self.call_params.get("max_tokens"),
+            # "top_k": self.call_params.get("top_k", 50),
+            # "top_p": self.call_params.get("top_p", 0.9),
+        }
+        return params
+
+    def log_info(self, message: str) -> None:
+        """Log a message to the console."""
+        # FIXME: The logger.info will not always log so for now I'm printing to stdout
+        # remove and log properly.
+        if bool(self.config.get("verbose", False)):
+            print(message)
+
+    @abstractmethod
+    def dispatch_no_streaming(self, mtir: MTIR) -> CompletionResult:
+        """Dispatch the LLM call without streaming."""
+        raise NotImplementedError()
+
+    @abstractmethod
+    def dispatch_streaming(self, mtir: MTIR) -> Generator[str, None, None]:
+        """Dispatch the LLM call with streaming."""
+        raise NotImplementedError()
+
+
+# -----------------------------------------------------------------------------
+# Mock LLM Connector
+# -----------------------------------------------------------------------------
+
+
+class MockLLMConnector(LLMConnector):
+    """LLM Connector for a mock LLM service that simulates responses."""
+
+    @override
+    def dispatch_no_streaming(self, mtir: MTIR) -> CompletionResult:
+        """Dispatch the mock LLM call with the given request."""
+        output = self.config["outputs"].pop(0)  # type: ignore
+
+        if isinstance(output, MockToolCall):
+            self.log_info(
+                f"Mock LLM call completed with tool call:\n{output.to_tool_call()}"
+            )
+            return CompletionResult(
+                output=None,
+                tool_calls=[output.to_tool_call()],
+            )
+
+        self.log_info(f"Mock LLM call completed with response:\n{output}")
+
+        return CompletionResult(
+            output=output,
+            tool_calls=[],
+        )
+
+    @override
+    def dispatch_streaming(self, mtir: MTIR) -> Generator[str, None, None]:
+        """Dispatch the mock LLM call with the given request."""
+        output = self.config["outputs"].pop(0)  # type: ignore
+        if mtir.stream:
+            while output:
+                chunk_len = random.randint(3, 10)
+                yield output[:chunk_len]  # Simulate token chunk
+                time.sleep(random.uniform(0.01, 0.05))  # Simulate network delay
+                output = output[chunk_len:]
+
+
+# -----------------------------------------------------------------------------
+# LiteLLM Connector
+# -----------------------------------------------------------------------------
+
+
+class LiteLLMConnector(LLMConnector):
+    """LLM Connector for LiteLLM, a lightweight wrapper around OpenAI API."""
+
+    def __init__(self, proxy: bool, model_name: str, **kwargs: object) -> None:
+        """Initialize the LiteLLM connector."""
+        super().__init__(model_name, **kwargs)
+        self.proxy = proxy
+
+        # Every litellm call will be logged to the tty and that pollutes the output.
+        # When there is a by llm() call in the jaclang.
+        logging.getLogger("httpx").setLevel(logging.WARNING)
+        _disable_debugging()
+        litellm.drop_params = True
+
+    @override
+    def dispatch_no_streaming(self, mtir: MTIR) -> CompletionResult:
+        """Dispatch the LLM call without streaming."""
+        # Construct the parameters for the LLM call
+        params = self.make_model_params(mtir)
+
+        # Call the LiteLLM API
+        self.log_info(f"Calling LLM: {self.model_name} with params:\n{params}")
+        if self.proxy:
+            client = OpenAI(
+                base_url=params.pop("api_base", "htpp://localhost:4000"),
+                api_key=params.pop("api_key"),
+            )
+            response = client.chat.completions.create(**params)
+        else:
+            response = litellm.completion(**params)
+
+        # Output format:
+        # https://docs.litellm.ai/docs/#response-format-openai-format
+        #
+        # TODO: Handle stream output (type ignoring stream response)
+        message: LiteLLMMessage = response.choices[0].message  # type: ignore
+        mtir.add_message(message)
+
+        output_content: str = message.content  # type: ignore
+        self.log_info(f"LLM call completed with response:\n{output_content}")
+        output_value = mtir.parse_response(output_content)
+
+        tool_calls: list[ToolCall] = []
+        for tool_call in message.tool_calls or []:  # type: ignore
+            if tool := mtir.get_tool(tool_call["function"]["name"]):
+                args_json = json.loads(tool_call["function"]["arguments"])
+                args = tool.parse_arguments(args_json)
+                tool_calls.append(
+                    ToolCall(call_id=tool_call["id"], tool=tool, args=args)
+                )
+            else:
+                raise RuntimeError(
+                    f"Attempted to call tool: '{tool_call['function']['name']}' which was not present."
+                )
+
+        return CompletionResult(
+            output=output_value,
+            tool_calls=tool_calls,
+        )
+
+    @override
+    def dispatch_streaming(self, mtir: MTIR) -> Generator[str, None, None]:
+        """Dispatch the LLM call with streaming."""
+        # Construct the parameters for the LLM call
+        params = self.make_model_params(mtir)
+
+        # Call the LiteLLM API
+        self.log_info(f"Calling LLM: {self.model_name} with params:\n{params}")
+        if self.proxy:
+            client = OpenAI(
+                base_url=params.pop("api_base"),
+                api_key=params.pop("api_key"),
+            )
+
+            # Call the LiteLLM API
+            response = client.chat.completions.create(**params, stream=True)
+        else:
+            response = litellm.completion(**params, stream=True)  # type: ignore
+
+        for chunk in response:
+            if chunk.choices and chunk.choices[0].delta:
+                delta = chunk.choices[0].delta
+                yield delta.content or ""