local-llm-kit 0.1.0__tar.gz

local_llm_kit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Utkarsh Rajput (GitHub: 1Utkarsh1)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

local_llm_kit-0.1.0/MANIFEST.in

@@ -0,0 +1,9 @@
+include LICENSE
+include README.md
+include requirements.txt
+recursive-include examples *
+recursive-include tests *
+global-exclude __pycache__
+global-exclude *.py[cod]
+global-exclude *.so
+global-exclude .DS_Store 

local_llm_kit-0.1.0/PKG-INFO

@@ -0,0 +1,249 @@
+Metadata-Version: 2.4
+Name: local-llm-kit
+Version: 0.1.0
+Summary: OpenAI-like interface for local LLMs
+Home-page: https://github.com/1Utkarsh1/local-llm-kit
+Author: Utkarsh Rajput
+Author-email: info@example.com
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typing-extensions>=4.0.0
+Provides-Extra: transformers
+Requires-Dist: torch>=1.12.0; extra == "transformers"
+Requires-Dist: transformers>=4.25.0; extra == "transformers"
+Requires-Dist: accelerate>=0.16.0; extra == "transformers"
+Provides-Extra: llamacpp
+Requires-Dist: llama-cpp-python>=0.1.77; extra == "llamacpp"
+Provides-Extra: all
+Requires-Dist: torch>=1.12.0; extra == "all"
+Requires-Dist: transformers>=4.25.0; extra == "all"
+Requires-Dist: accelerate>=0.16.0; extra == "all"
+Requires-Dist: llama-cpp-python>=0.1.77; extra == "all"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Local LLM Kit
+
+An OpenAI-like interface for local Large Language Models. This package allows you to interact with local LLMs using an API similar to OpenAI's, including support for chat completions, function calling, streaming responses, and more.
+
+**Created by Utkarsh Rajput (GitHub: [1Utkarsh1](https://github.com/1Utkarsh1))** - A powerful tool for working with local language models with all the convenience of the OpenAI API.
+
+## Features
+
+- 🗣️ **Chat and Completion API**: Similar to OpenAI's API
+- 🧩 **Multiple Backend Support**: Works with Hugging Face Transformers and llama.cpp
+- 🛠️ **Function Calling**: Register Python functions that LLMs can call
+- 📱 **Streaming Support**: Stream responses token by token
+- 🧠 **Memory Management**: Auto-truncation and context management
+- 📊 **Logprobs**: Get token probabilities for generations
+- 📝 **Prompt Formatting**: Supports various model template formats (Llama, Mistral, etc.)
+- 🌐 **JSON Mode**: Enforce structured JSON output
+
+## Installation
+
+### Basic Installation
+
+```bash
+pip install local-llm-kit
+```
+
+### With Backend Support
+
+```bash
+# For Hugging Face Transformers support
+pip install "local-llm-kit[transformers]"
+
+# For llama.cpp support
+pip install "local-llm-kit[llamacpp]"
+
+# For all backends
+pip install "local-llm-kit[all]"
+```
+
+## Quick Start
+
+### Chat with a Model
+
+```python
+from local_llm_kit import chat
+
+response = chat(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me about the solar system."}
+    ],
+    model_path="path/to/your/model",  # Local model path or HF model name
+    temperature=0.7,
+    max_tokens=512
+)
+
+print(response["choices"][0]["message"]["content"])
+```
+
+### Text Completion
+
+```python
+from local_llm_kit import complete
+
+response = complete(
+    prompt="The solar system consists of",
+    model_path="path/to/your/model",
+    temperature=0.7,
+    max_tokens=512
+)
+
+print(response["choices"][0]["text"])
+```
+
+### Function Calling
+
+```python
+from local_llm_kit import LLM
+
+# Define a function with its schema
+def get_weather(location: str, unit: str = "celsius"):
+    """Get the weather for a location."""
+    # In a real app, this would call a weather API
+    return {"temperature": 22, "unit": unit, "description": f"Sunny in {location}"}
+
+weather_function = {
+    "name": "get_weather",
+    "description": "Get the current weather for a location",
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "location": {
+                "type": "string",
+                "description": "The city and state, e.g. San Francisco, CA"
+            },
+            "unit": {
+                "type": "string",
+                "enum": ["celsius", "fahrenheit"],
+                "description": "The unit of temperature"
+            }
+        },
+        "required": ["location"]
+    }
+}
+
+# Create LLM instance
+llm = LLM(model_path="path/to/your/model")
+
+# Register the function
+llm.add_function(
+    name="get_weather",
+    schema=weather_function,
+    implementation=get_weather
+)
+
+# Chat with function calling
+response = llm.chat(
+    messages=[
+        {"role": "user", "content": "What's the weather like in Paris?"}
+    ],
+    # The function will be automatically called if the model chooses to use it
+)
+
+print(response["choices"][0]["message"]["content"])
+```
+
+### Streaming Responses
+
+```python
+from local_llm_kit import chat
+
+for chunk in chat(
+    messages=[
+        {"role": "user", "content": "Write a short poem about nature."}
+    ],
+    model_path="path/to/your/model",
+    stream=True
+):
+    # Process each token as it's generated
+    if "choices" in chunk and chunk["choices"] and "delta" in chunk["choices"][0]:
+        delta = chunk["choices"][0]["delta"]
+        if "content" in delta:
+            print(delta["content"], end="", flush=True)
+```
+
+## Command Line Interface
+
+Local LLM Kit includes a CLI for easy interaction with models.
+
+### Interactive Chat
+
+```bash
+local-llm-kit chat --model path/to/your/model --system "You are a helpful assistant."
+```
+
+### Text Completion
+
+```bash
+local-llm-kit complete --model path/to/your/model --prompt "Once upon a time,"
+```
+
+## Advanced Usage
+
+### Using Different Backends
+
+```python
+from local_llm_kit import LLM
+
+# Using Transformers backend
+llm_transformers = LLM(
+    model_path="mistralai/Mistral-7B-Instruct-v0.1",
+    backend="transformers",
+    backend_kwargs={"device": "cuda", "torch_dtype": "float16"}
+)
+
+# Using llama.cpp backend
+llm_llamacpp = LLM(
+    model_path="path/to/model.gguf",
+    backend="llamacpp",
+    backend_kwargs={"n_gpu_layers": -1, "n_ctx": 4096}
+)
+```
+
+### JSON Mode
+
+```python
+from local_llm_kit import chat
+
+response = chat(
+    messages=[
+        {"role": "user", "content": "Generate a JSON list of 3 planets with their diameter."}
+    ],
+    model_path="path/to/your/model",
+    format="json"
+)
+
+import json
+planets = json.loads(response["choices"][0]["message"]["content"])
+print(planets)
+```
+
+## License
+
+MIT License
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. 

local_llm_kit-0.1.0/README.md

@@ -0,0 +1,207 @@
+# Local LLM Kit
+
+An OpenAI-like interface for local Large Language Models. This package allows you to interact with local LLMs using an API similar to OpenAI's, including support for chat completions, function calling, streaming responses, and more.
+
+**Created by Utkarsh Rajput (GitHub: [1Utkarsh1](https://github.com/1Utkarsh1))** - A powerful tool for working with local language models with all the convenience of the OpenAI API.
+
+## Features
+
+- 🗣️ **Chat and Completion API**: Similar to OpenAI's API
+- 🧩 **Multiple Backend Support**: Works with Hugging Face Transformers and llama.cpp
+- 🛠️ **Function Calling**: Register Python functions that LLMs can call
+- 📱 **Streaming Support**: Stream responses token by token
+- 🧠 **Memory Management**: Auto-truncation and context management
+- 📊 **Logprobs**: Get token probabilities for generations
+- 📝 **Prompt Formatting**: Supports various model template formats (Llama, Mistral, etc.)
+- 🌐 **JSON Mode**: Enforce structured JSON output
+
+## Installation
+
+### Basic Installation
+
+```bash
+pip install local-llm-kit
+```
+
+### With Backend Support
+
+```bash
+# For Hugging Face Transformers support
+pip install "local-llm-kit[transformers]"
+
+# For llama.cpp support
+pip install "local-llm-kit[llamacpp]"
+
+# For all backends
+pip install "local-llm-kit[all]"
+```
+
+## Quick Start
+
+### Chat with a Model
+
+```python
+from local_llm_kit import chat
+
+response = chat(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me about the solar system."}
+    ],
+    model_path="path/to/your/model",  # Local model path or HF model name
+    temperature=0.7,
+    max_tokens=512
+)
+
+print(response["choices"][0]["message"]["content"])
+```
+
+### Text Completion
+
+```python
+from local_llm_kit import complete
+
+response = complete(
+    prompt="The solar system consists of",
+    model_path="path/to/your/model",
+    temperature=0.7,
+    max_tokens=512
+)
+
+print(response["choices"][0]["text"])
+```
+
+### Function Calling
+
+```python
+from local_llm_kit import LLM
+
+# Define a function with its schema
+def get_weather(location: str, unit: str = "celsius"):
+    """Get the weather for a location."""
+    # In a real app, this would call a weather API
+    return {"temperature": 22, "unit": unit, "description": f"Sunny in {location}"}
+
+weather_function = {
+    "name": "get_weather",
+    "description": "Get the current weather for a location",
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "location": {
+                "type": "string",
+                "description": "The city and state, e.g. San Francisco, CA"
+            },
+            "unit": {
+                "type": "string",
+                "enum": ["celsius", "fahrenheit"],
+                "description": "The unit of temperature"
+            }
+        },
+        "required": ["location"]
+    }
+}
+
+# Create LLM instance
+llm = LLM(model_path="path/to/your/model")
+
+# Register the function
+llm.add_function(
+    name="get_weather",
+    schema=weather_function,
+    implementation=get_weather
+)
+
+# Chat with function calling
+response = llm.chat(
+    messages=[
+        {"role": "user", "content": "What's the weather like in Paris?"}
+    ],
+    # The function will be automatically called if the model chooses to use it
+)
+
+print(response["choices"][0]["message"]["content"])
+```
+
+### Streaming Responses
+
+```python
+from local_llm_kit import chat
+
+for chunk in chat(
+    messages=[
+        {"role": "user", "content": "Write a short poem about nature."}
+    ],
+    model_path="path/to/your/model",
+    stream=True
+):
+    # Process each token as it's generated
+    if "choices" in chunk and chunk["choices"] and "delta" in chunk["choices"][0]:
+        delta = chunk["choices"][0]["delta"]
+        if "content" in delta:
+            print(delta["content"], end="", flush=True)
+```
+
+## Command Line Interface
+
+Local LLM Kit includes a CLI for easy interaction with models.
+
+### Interactive Chat
+
+```bash
+local-llm-kit chat --model path/to/your/model --system "You are a helpful assistant."
+```
+
+### Text Completion
+
+```bash
+local-llm-kit complete --model path/to/your/model --prompt "Once upon a time,"
+```
+
+## Advanced Usage
+
+### Using Different Backends
+
+```python
+from local_llm_kit import LLM
+
+# Using Transformers backend
+llm_transformers = LLM(
+    model_path="mistralai/Mistral-7B-Instruct-v0.1",
+    backend="transformers",
+    backend_kwargs={"device": "cuda", "torch_dtype": "float16"}
+)
+
+# Using llama.cpp backend
+llm_llamacpp = LLM(
+    model_path="path/to/model.gguf",
+    backend="llamacpp",
+    backend_kwargs={"n_gpu_layers": -1, "n_ctx": 4096}
+)
+```
+
+### JSON Mode
+
+```python
+from local_llm_kit import chat
+
+response = chat(
+    messages=[
+        {"role": "user", "content": "Generate a JSON list of 3 planets with their diameter."}
+    ],
+    model_path="path/to/your/model",
+    format="json"
+)
+
+import json
+planets = json.loads(response["choices"][0]["message"]["content"])
+print(planets)
+```
+
+## License
+
+MIT License
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. 

local_llm_kit-0.1.0/examples/__init__.py

@@ -0,0 +1,3 @@
+"""
+Example scripts for local_llm_kit.
+""" 

local_llm_kit-0.1.0/examples/simple_chat.py

@@ -0,0 +1,66 @@
+"""
+Simple example script for local_llm_kit chat.
+"""
+import argparse
+import sys
+
+from local_llm_kit import chat
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Simple chat with a local LLM")
+    parser.add_argument("--model", "-m", required=True, help="Path to the model or model name")
+    parser.add_argument("--backend", "-b", choices=["transformers", "llamacpp"], 
+                        help="Backend to use (default: auto-detect)")
+    args = parser.parse_args()
+    
+    # Initial message
+    print(f"Simple chat with {args.model}")
+    print("Type 'exit' or 'quit' to end the conversation.")
+    print()
+    
+    # Chat loop
+    messages = [
+        {"role": "system", "content": "You are a helpful assistant."}
+    ]
+    
+    try:
+        while True:
+            # Get user input
+            user_input = input("You: ")
+            if user_input.lower() in ["exit", "quit"]:
+                print("Goodbye!")
+                break
+                
+            # Add user message
+            messages.append({"role": "user", "content": user_input})
+            
+            # Generate and display response
+            try:
+                print("Assistant: ", end="", flush=True)
+                response_text = ""
+                
+                # Stream the response token by token
+                for chunk in chat(
+                    messages=messages,
+                    model_path=args.model,
+                    backend=args.backend,
+                    stream=True
+                ):
+                    # Extract and print chunk text
+                    if "choices" in chunk and chunk["choices"] and "delta" in chunk["choices"][0]:
+                        delta = chunk["choices"][0]["delta"]
+                        if "content" in delta and delta["content"]:
+                            print(delta["content"], end="", flush=True)
+                            response_text += delta["content"]
+                
+                print()  # Add newline after response
+                
+                # Add assistant response to history
+                messages.append({"role": "assistant", "content": response_text})
+                
+            except Exception as e:
+                print(f"\nError: {e}")
+                
+    except KeyboardInterrupt:
+        print("\nGoodbye!")
+        sys.exit(0) 

local_llm_kit-0.1.0/local_llm_kit/__init__.py

@@ -0,0 +1,11 @@
+"""
+local_llm_kit - OpenAI-like interface for local LLMs
+"""
+
+__version__ = "0.1.0"
+
+from .llm import LLM
+from .chat import chat, complete
+from .function_calling import add_function
+
+__all__ = ["LLM", "chat", "complete", "add_function"] 

local_llm_kit-0.1.0/local_llm_kit/backends/__init__.py

@@ -0,0 +1,9 @@
+"""
+Model backends for inference.
+"""
+
+from .base import BaseBackend
+from .transformers import TransformersBackend
+from .llamacpp import LlamaCppBackend
+
+__all__ = ["BaseBackend", "TransformersBackend", "LlamaCppBackend"] 

local_llm_kit-0.1.0/local_llm_kit/backends/base.py

@@ -0,0 +1,113 @@
+"""
+Base class for model backends.
+"""
+import time
+from abc import ABC, abstractmethod
+from typing import Dict, Any, Iterator, Optional
+
+
+class BaseBackend(ABC):
+    """
+    Abstract base class for model backends.
+    
+    All backend implementations must inherit from this class and
+    implement the required methods.
+    """
+    
+    @abstractmethod
+    def generate(
+        self, 
+        prompt: str, 
+        temperature: float = 0.7,
+        max_new_tokens: int = 512,
+        top_p: float = 0.95,
+        top_k: int = 40,
+        repetition_penalty: float = 1.1,
+        stream: bool = False,
+        logprobs: bool = False,
+        top_logprobs: Optional[int] = None,
+        **kwargs
+    ) -> Dict[str, Any]:
+        """
+        Generate text completion for a prompt.
+        
+        Args:
+            prompt: Input text to complete
+            temperature: Sampling temperature (higher = more random)
+            max_new_tokens: Maximum number of tokens to generate
+            top_p: Top-p sampling parameter
+            top_k: Top-k sampling parameter
+            repetition_penalty: Penalty for repetition
+            stream: Whether to stream the response (should be False for this method)
+            logprobs: Whether to return log probabilities
+            top_logprobs: Number of top tokens to return logprobs for
+            **kwargs: Additional backend-specific parameters
+            
+        Returns:
+            Dictionary with at least a "text" key containing the generated text
+        """
+        pass
+    
+    @abstractmethod
+    def generate_stream(
+        self, 
+        prompt: str,
+        temperature: float = 0.7,
+        max_new_tokens: int = 512,
+        top_p: float = 0.95,
+        top_k: int = 40,
+        repetition_penalty: float = 1.1,
+        logprobs: bool = False,
+        top_logprobs: Optional[int] = None,
+        **kwargs
+    ) -> Iterator[Dict[str, Any]]:
+        """
+        Stream text completion for a prompt.
+        
+        Args:
+            prompt: Input text to complete
+            temperature: Sampling temperature (higher = more random)
+            max_new_tokens: Maximum number of tokens to generate
+            top_p: Top-p sampling parameter
+            top_k: Top-k sampling parameter
+            repetition_penalty: Penalty for repetition
+            logprobs: Whether to return log probabilities
+            top_logprobs: Number of top tokens to return logprobs for
+            **kwargs: Additional backend-specific parameters
+            
+        Yields:
+            Dictionaries with at least a "text" key containing a chunk of generated text
+        """
+        pass
+    
+    @abstractmethod
+    def get_context_window(self) -> int:
+        """
+        Get the context window size of the model.
+        
+        Returns:
+            Maximum context window size in tokens
+        """
+        pass
+    
+    @abstractmethod
+    def count_tokens(self, text: str) -> int:
+        """
+        Count the number of tokens in a text.
+        
+        Args:
+            text: Input text
+            
+        Returns:
+            Token count
+        """
+        pass
+    
+    def get_timestamp(self) -> int:
+        """
+        Get current timestamp.
+        
+        Returns:
+            Current timestamp in seconds
+        """
+        return int(time.time())