inference-proxy 0.2.0__tar.gz → 0.2.2__tar.gz

inference_proxy-0.2.2/PKG-INFO

@@ -0,0 +1,288 @@
+Metadata-Version: 2.3
+Name: inference-proxy
+Version: 0.2.2
+Summary: "Inference Proxy" is OpenAI-compatible http proxy server for inferencing various LLMs capable of working with Google, Anthropic, OpenAI APIs, local PyTorch inference, etc.
+License: MIT License
+         
+         Copyright (c) 2025 Vitalii Stepanenko
+         
+         Permission is hereby granted, free of charge, to any person obtaining a copy
+         of this software and associated documentation files (the "Software"), to deal
+         in the Software without restriction, including without limitation the rights
+         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+         copies of the Software, and to permit persons to whom the Software is
+         furnished to do so, subject to the following conditions:
+         
+         The above copyright notice and this permission notice shall be included in all
+         copies or substantial portions of the Software.
+         
+         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+         SOFTWARE.
+Keywords: llm,large language models,ai,gpt,openai,proxy,http,proxy-server
+Author: Vitalii Stepanenko
+Author-email: mail@vitalii.in
+Maintainer: Vitalii Stepanenko
+Maintainer-email: mail@vitalii.in
+Requires-Python: >=3.10,<4
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: License :: OSI Approved :: MIT License
+Requires-Dist: ai-microcore (>=4.4.3,<4.5.0)
+Requires-Dist: fastapi (>=0.116.1,<0.117.0)
+Requires-Dist: typer (>=0.16.1)
+Requires-Dist: uvicorn (>=0.22.0)
+Project-URL: Source Code, https://github.com/Nayjest/lm-proxy
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://img.shields.io/github/license/Nayjest/lm-proxy?color=blue" alt="License">
+  <a href="https://pypi.org/project/lm-proxy/"><img src="https://img.shields.io/pypi/v/lm-proxy?color=blue" alt="PyPI"></a>
+  <a href="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
+  <a href="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml/badge.svg" alt="Code Style"></a>
+</p>
+
+# Inference Proxy
+
+**Inference Proxy** is an OpenAI-compatible HTTP proxy server for various Large Language Models (LLMs) inference. 
+It provides a unified interface for working with different AI providers through a single API endpoint that follows the OpenAI format.
+Stream like OpenAI, authenticate with your own API keys, and keep clients unchanged.
+## ✨ Features
+
+- **Provider Agnostic**: Connect to OpenAI, Anthropic, Google AI, local models, and more using a single API
+- **Unified Interface**: Access all models through the standard OpenAI API format
+- **Dynamic Routing**: Route requests to different LLM providers based on model name patterns
+- **Stream Support**: Full streaming support for real-time responses
+- **API Key Management**: Configurable API key validation and access control
+- **Easy Configuration**: Simple TOML configuration files for setup
+
+## 🚀 Getting Started
+
+### Installation
+
+```bash
+pip install inference-proxy
+```
+
+### Quick Start
+
+1. Create a `config.toml` file:
+
+```toml
+host = "0.0.0.0"
+port = 8000
+
+[connections]
+[connections.openai]
+api_type = "open_ai"
+api_base = "https://api.openai.com/v1/"
+api_key = "env:OPENAI_API_KEY"
+
+[connections.anthropic]
+api_type = "anthropic"
+api_key = "env:ANTHROPIC_API_KEY"
+
+[routing]
+"gpt*" = "openai.*"
+"claude*" = "anthropic.*"
+"*" = "openai.gpt-3.5-turbo"
+
+[groups.default]
+api_keys = ["YOUR_API_KEY_HERE"]
+```
+
+2. Start the server:
+
+```bash
+inference-proxy
+```
+
+3. Use it with any OpenAI-compatible client:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="YOUR_API_KEY_HERE",
+    base_url="http://localhost:8000/v1"
+)
+
+completion = client.chat.completions.create(
+    model="gpt-5",  # This will be routed to OpenAI based on config
+    messages=[{"role": "user", "content": "Hello, world!"}]
+)
+print(completion.choices[0].message.content)
+```
+
+Or use the same endpoint with Claude models:
+
+```python
+completion = client.chat.completions.create(
+    model="claude-opus-4-1-20250805",  # This will be routed to Anthropic based on config
+    messages=[{"role": "user", "content": "Hello, world!"}]
+)
+```
+
+## 📝 Configuration
+
+Inference Proxy is configured through a TOML file that specifies connections, routing rules, and access control.
+
+### Basic Structure
+
+```toml
+host = "0.0.0.0"  # Interface to bind to
+port = 8000       # Port to listen on
+dev_autoreload = false  # Enable for development
+
+# API key validation function (optional)
+check_api_key = "lm_proxy.core.check_api_key"
+
+# LLM Provider Connections
+[connections]
+
+[connections.openai]
+api_type = "open_ai"
+api_base = "https://api.openai.com/v1/"
+api_key = "env:OPENAI_API_KEY"
+
+[connections.google]
+api_type = "google_ai_studio"
+api_key = "env:GOOGLE_API_KEY"
+
+# Routing rules (model_pattern = "connection.model")
+[routing]
+"gpt*" = "openai.*"     # Route all GPT models to OpenAI
+"claude*" = "anthropic.*"  # Route all Claude models to Anthropic
+"gemini*" = "google.*"  # Route all Gemini models to Google
+"*" = "openai.gpt-3.5-turbo"  # Default fallback
+
+# Access control groups
+[groups.default]
+api_keys = [
+    "KEY1",
+    "KEY2"
+]
+```
+
+### Environment Variables
+
+You can use environment variables in your configuration file by prefixing values with `env:`:
+
+```toml
+[connections.openai]
+api_key = "env:OPENAI_API_KEY"
+```
+
+Load these from a `.env` file or set them in your environment before starting the server.
+
+## 🔌 API Usage
+
+Inference Proxy implements the OpenAI chat completions API endpoint. You can use any OpenAI-compatible client to interact with it.
+
+### Endpoint
+
+```
+POST /v1/chat/completions
+```
+
+### Request Format
+
+```json
+{
+  "model": "gpt-3.5-turbo",
+  "messages": [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "What is the capital of France?"}
+  ],
+  "temperature": 0.7,
+  "stream": false
+}
+```
+
+### Response Format
+
+```json
+{
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": "The capital of France is Paris."
+      },
+      "finish_reason": "stop"
+    }
+  ]
+}
+```
+
+## 🛠️ Advanced Usage
+
+### Custom API Key Validation
+
+You can implement your own API key validation function:
+
+```python
+# my_validators.py
+def validate_api_key(api_key: str) -> str | None:
+    """
+    Validate an API key and return the group name if valid.
+    
+    Args:
+        api_key: The API key to validate
+        
+    Returns:
+        The name of the group if valid, None otherwise
+    """
+    if api_key == "secret-key":
+        return "admin"
+    elif api_key.startswith("user-"):
+        return "users"
+    return None
+```
+
+Then reference it in your config:
+
+```toml
+check_api_key = "my_validators.validate_api_key"
+```
+
+### Dynamic Model Routing
+
+The routing section allows flexible pattern matching with wildcards:
+
+```toml
+[routing]
+"gpt-4*" = "openai.gpt-4"           # Route gpt-4 requests to OpenAI GPT-4
+"gpt-3.5*" = "openai.gpt-3.5-turbo" # Route gpt-3.5 requests to OpenAI
+"claude*" = "anthropic.*"           # Pass model name as-is to Anthropic
+"gemini*" = "google.*"              # Pass model name as-is to Google
+"custom*" = "local.llama-7b"        # Map any "custom*" to a specific local model
+"*" = "openai.gpt-3.5-turbo"        # Default fallback for unmatched models
+```
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add some amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+© 2025 Vitalii Stepanenko
+

inference_proxy-0.2.2/README.md

@@ -0,0 +1,241 @@
+<p align="center">
+  <img src="https://img.shields.io/github/license/Nayjest/lm-proxy?color=blue" alt="License">
+  <a href="https://pypi.org/project/lm-proxy/"><img src="https://img.shields.io/pypi/v/lm-proxy?color=blue" alt="PyPI"></a>
+  <a href="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
+  <a href="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml/badge.svg" alt="Code Style"></a>
+</p>
+
+# Inference Proxy
+
+**Inference Proxy** is an OpenAI-compatible HTTP proxy server for various Large Language Models (LLMs) inference. 
+It provides a unified interface for working with different AI providers through a single API endpoint that follows the OpenAI format.
+Stream like OpenAI, authenticate with your own API keys, and keep clients unchanged.
+## ✨ Features
+
+- **Provider Agnostic**: Connect to OpenAI, Anthropic, Google AI, local models, and more using a single API
+- **Unified Interface**: Access all models through the standard OpenAI API format
+- **Dynamic Routing**: Route requests to different LLM providers based on model name patterns
+- **Stream Support**: Full streaming support for real-time responses
+- **API Key Management**: Configurable API key validation and access control
+- **Easy Configuration**: Simple TOML configuration files for setup
+
+## 🚀 Getting Started
+
+### Installation
+
+```bash
+pip install inference-proxy
+```
+
+### Quick Start
+
+1. Create a `config.toml` file:
+
+```toml
+host = "0.0.0.0"
+port = 8000
+
+[connections]
+[connections.openai]
+api_type = "open_ai"
+api_base = "https://api.openai.com/v1/"
+api_key = "env:OPENAI_API_KEY"
+
+[connections.anthropic]
+api_type = "anthropic"
+api_key = "env:ANTHROPIC_API_KEY"
+
+[routing]
+"gpt*" = "openai.*"
+"claude*" = "anthropic.*"
+"*" = "openai.gpt-3.5-turbo"
+
+[groups.default]
+api_keys = ["YOUR_API_KEY_HERE"]
+```
+
+2. Start the server:
+
+```bash
+inference-proxy
+```
+
+3. Use it with any OpenAI-compatible client:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="YOUR_API_KEY_HERE",
+    base_url="http://localhost:8000/v1"
+)
+
+completion = client.chat.completions.create(
+    model="gpt-5",  # This will be routed to OpenAI based on config
+    messages=[{"role": "user", "content": "Hello, world!"}]
+)
+print(completion.choices[0].message.content)
+```
+
+Or use the same endpoint with Claude models:
+
+```python
+completion = client.chat.completions.create(
+    model="claude-opus-4-1-20250805",  # This will be routed to Anthropic based on config
+    messages=[{"role": "user", "content": "Hello, world!"}]
+)
+```
+
+## 📝 Configuration
+
+Inference Proxy is configured through a TOML file that specifies connections, routing rules, and access control.
+
+### Basic Structure
+
+```toml
+host = "0.0.0.0"  # Interface to bind to
+port = 8000       # Port to listen on
+dev_autoreload = false  # Enable for development
+
+# API key validation function (optional)
+check_api_key = "lm_proxy.core.check_api_key"
+
+# LLM Provider Connections
+[connections]
+
+[connections.openai]
+api_type = "open_ai"
+api_base = "https://api.openai.com/v1/"
+api_key = "env:OPENAI_API_KEY"
+
+[connections.google]
+api_type = "google_ai_studio"
+api_key = "env:GOOGLE_API_KEY"
+
+# Routing rules (model_pattern = "connection.model")
+[routing]
+"gpt*" = "openai.*"     # Route all GPT models to OpenAI
+"claude*" = "anthropic.*"  # Route all Claude models to Anthropic
+"gemini*" = "google.*"  # Route all Gemini models to Google
+"*" = "openai.gpt-3.5-turbo"  # Default fallback
+
+# Access control groups
+[groups.default]
+api_keys = [
+    "KEY1",
+    "KEY2"
+]
+```
+
+### Environment Variables
+
+You can use environment variables in your configuration file by prefixing values with `env:`:
+
+```toml
+[connections.openai]
+api_key = "env:OPENAI_API_KEY"
+```
+
+Load these from a `.env` file or set them in your environment before starting the server.
+
+## 🔌 API Usage
+
+Inference Proxy implements the OpenAI chat completions API endpoint. You can use any OpenAI-compatible client to interact with it.
+
+### Endpoint
+
+```
+POST /v1/chat/completions
+```
+
+### Request Format
+
+```json
+{
+  "model": "gpt-3.5-turbo",
+  "messages": [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "What is the capital of France?"}
+  ],
+  "temperature": 0.7,
+  "stream": false
+}
+```
+
+### Response Format
+
+```json
+{
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": "The capital of France is Paris."
+      },
+      "finish_reason": "stop"
+    }
+  ]
+}
+```
+
+## 🛠️ Advanced Usage
+
+### Custom API Key Validation
+
+You can implement your own API key validation function:
+
+```python
+# my_validators.py
+def validate_api_key(api_key: str) -> str | None:
+    """
+    Validate an API key and return the group name if valid.
+    
+    Args:
+        api_key: The API key to validate
+        
+    Returns:
+        The name of the group if valid, None otherwise
+    """
+    if api_key == "secret-key":
+        return "admin"
+    elif api_key.startswith("user-"):
+        return "users"
+    return None
+```
+
+Then reference it in your config:
+
+```toml
+check_api_key = "my_validators.validate_api_key"
+```
+
+### Dynamic Model Routing
+
+The routing section allows flexible pattern matching with wildcards:
+
+```toml
+[routing]
+"gpt-4*" = "openai.gpt-4"           # Route gpt-4 requests to OpenAI GPT-4
+"gpt-3.5*" = "openai.gpt-3.5-turbo" # Route gpt-3.5 requests to OpenAI
+"claude*" = "anthropic.*"           # Pass model name as-is to Anthropic
+"gemini*" = "google.*"              # Pass model name as-is to Google
+"custom*" = "local.llama-7b"        # Map any "custom*" to a specific local model
+"*" = "openai.gpt-3.5-turbo"        # Default fallback for unmatched models
+```
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add some amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+© 2025 Vitalii Stepanenko

{inference_proxy-0.2.0 → inference_proxy-0.2.2}/lm_proxy/__main__.py

@@ -1,5 +1,5 @@
-from .app import cli_app
-
-
-if __name__ == "__main__":
-    cli_app()
+from .app import cli_app
+
+
+if __name__ == "__main__":
+    cli_app()

{inference_proxy-0.2.0 → inference_proxy-0.2.2}/lm_proxy/app.py

@@ -20,6 +20,7 @@ def run_server(
         host=env.config.host,
         port=env.config.port,
         reload=env.config.dev_autoreload,
+        factory=True,
     )
 
 

inference_proxy-0.2.2/lm_proxy/bootstrap.py

@@ -0,0 +1,81 @@
+import sys
+import logging
+import inspect
+from datetime import datetime
+
+
+import microcore as mc
+from microcore import ui
+from microcore.configuration import get_bool_from_env
+from dotenv import load_dotenv
+
+from .config import Config
+
+
+def setup_logging(log_level: int = logging.INFO):
+    class CustomFormatter(logging.Formatter):
+        def format(self, record):
+            dt = datetime.fromtimestamp(record.created).strftime("%H:%M:%S")
+            message, level_name = record.getMessage(), record.levelname
+            if record.levelno == logging.WARNING:
+                message = mc.ui.yellow(message)
+                level_name = mc.ui.yellow(level_name)
+            if record.levelno >= logging.ERROR:
+                message = mc.ui.red(message)
+                level_name = mc.ui.red(level_name)
+            return f"{dt} {level_name}: {message}"
+
+    handler = logging.StreamHandler()
+    handler.setFormatter(CustomFormatter())
+    logging.basicConfig(level=log_level, handlers=[handler])
+
+
+class Env:
+    config: Config
+    connections: dict[str, mc.types.LLMAsyncFunctionType]
+    debug: bool
+
+    @staticmethod
+    def init(config: Config | str, debug: bool = False):
+        env.debug = debug
+
+        if isinstance(config, Config):
+            env.config = config
+        elif isinstance(config, str):
+            env.config = Config.load(config)
+        else:
+            raise ValueError("config must be a string (file path) or Config instance")
+
+        # initialize connections
+        env.connections = dict()
+        for conn_name, conn_config in env.config.connections.items():
+            logging.info(f"Initializing '{conn_name}' LLM proxy connection...")
+            try:
+                if inspect.iscoroutinefunction(conn_config):
+                    env.connections[conn_name] = conn_config
+                else:
+                    mc.configure(
+                        **conn_config,
+                        EMBEDDING_DB_TYPE=mc.EmbeddingDbType.NONE
+                    )
+                    env.connections[conn_name] = mc.env().llm_async_function
+            except mc.LLMConfigError as e:
+                raise ValueError(f"Error in configuration for connection '{conn_name}': {e}")
+
+        logging.info(f"Done initializing {len(env.connections)} connections.")
+
+
+env = Env()
+
+
+def bootstrap(config: str | Config = 'config.toml'):
+    load_dotenv('.env', override=True)
+    debug = '--debug' in sys.argv or get_bool_from_env('LM_PROXY_DEBUG', False)
+    setup_logging(logging.DEBUG if debug else logging.INFO)
+    mc.logging.LoggingConfig.OUTPUT_METHOD = logging.info
+    logging.info(
+        f"Bootstrapping {ui.yellow('lm_proxy')} "
+        f"using configuration: {'dynamic' if isinstance(config, Config) else ui.blue(config)} "
+        f"{'[DEBUG: ON]' if debug else ''}..."
+    )
+    Env.init(config, debug=debug)

{inference_proxy-0.2.0 → inference_proxy-0.2.2}/lm_proxy/config.py

@@ -7,7 +7,7 @@ from typing import Union, Callable
 import tomllib
 import importlib.util
 
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, ConfigDict
 from microcore.utils import resolve_callable
 
 
@@ -24,18 +24,17 @@ class Group(BaseModel):
 
 class Config(BaseModel):
     """Main configuration model matching config.toml structure."""
+    model_config = ConfigDict(extra="forbid")
     enabled: bool = True
     host: str = "0.0.0.0"
     port: int = 8000
     dev_autoreload: bool = False
     connections: dict[str, Union[dict, Callable]]
     routing: dict[str, str] = Field(default_factory=dict)
+    """ model_name_pattern* => connection_name.< model | * >, example: {"gpt-*": "oai.*"} """
     groups: dict[str, Group] = Field(default_factory=dict)
     check_api_key: Union[str, Callable] = Field(default="lm_proxy.core.check_api_key")
 
-    class Config:
-        extra = "forbid"
-
     def __init__(self, **data):
         super().__init__(**data)
         self.check_api_key = resolve_callable(self.check_api_key)
@@ -59,10 +58,11 @@ class Config(BaseModel):
             config_module = importlib.util.module_from_spec(spec)
             spec.loader.exec_module(config_module)
             return config_module.config
-
         elif config_path.endswith(".toml"):
             with open(config_path, "rb") as f:
                 config_data = tomllib.load(f)
+        else:
+            raise ValueError(f"Unsupported configuration file extension: {config_path}")
 
         # Process environment variables in api_key fields
         for conn_name, conn_config in config_data.get("connections", {}).items():

{inference_proxy-0.2.0 → inference_proxy-0.2.2}/lm_proxy/core.py

@@ -1,204 +1,204 @@
-import asyncio
-import fnmatch
-import json
-import logging
-import secrets
-import time
-from typing import List, Optional
-
-import microcore as mc
-from fastapi import HTTPException
-from pydantic import BaseModel
-from starlette.requests import Request
-from starlette.responses import JSONResponse, Response, StreamingResponse
-
-from .bootstrap import env
-from .config import Config, Group
-
-
-class ChatCompletionRequest(BaseModel):
-    model: str
-    messages: List[mc.Msg]
-    stream: Optional[bool] = None
-    max_tokens: Optional[int] = None
-    temperature: Optional[float] = None
-    top_p: Optional[float] = None
-    n: Optional[int] = None
-    stop: Optional[List[str]] = None
-    presence_penalty: Optional[float] = None
-    frequency_penalty: Optional[float] = None
-    user: Optional[str] = None
-
-
-def resolve_connection_and_model(config: Config, external_model: str) -> tuple[str, str]:
-    for model_match, rule in config.routing.items():
-        if fnmatch.fnmatchcase(external_model, model_match):
-            connection_name, model_part = rule.split(".", 1)
-            if connection_name not in config.connections:
-                raise ValueError(
-                    f"Routing selected unknown connection '{connection_name}'. "
-                    f"Defined connections: {', '.join(config.connections.keys()) or '(none)'}"
-                )
-
-            resolved_model = external_model if model_part == "*" else model_part
-            return connection_name, resolved_model
-
-    raise ValueError(
-        f"No routing rule matched model '{external_model}'. "
-        "Add a catch-all rule like \"*\" = \"openai.gpt-3.5-turbo\" if desired."
-    )
-
-
-async def process_stream(async_llm_func, prompt, llm_params):
-    queue = asyncio.Queue()
-    stream_id = f"chatcmpl-{secrets.token_hex(12)}"
-    created = int(time.time())
-
-    async def callback(chunk):
-        await queue.put(chunk)
-
-    def make_chunk(delta=None, content=None, finish_reason=None, error=None) -> str:
-        if delta is None:
-            delta = dict(content=str(content)) if content is not None else dict()
-        obj = {
-            "id": stream_id,
-            "object": "chat.completion.chunk",
-            "created": created,
-            "choices": [{"index": 0, "delta": delta}],
-        }
-        if error is not None:
-            obj['error'] = {'message': str(error), 'type': type(error).__name__}
-            if finish_reason is None:
-                finish_reason = 'error'
-        if finish_reason is not None:
-            obj['choices'][0]['finish_reason'] = finish_reason
-        return "data: " + json.dumps(obj) + "\n\n"
-
-    task = asyncio.create_task(
-        async_llm_func(prompt, **llm_params, callback=callback)
-    )
-
-    try:
-        # Initial chunk: role
-        yield make_chunk(delta={'role': 'assistant'})
-
-        while not task.done():
-            try:
-                block = await asyncio.wait_for(queue.get(), timeout=0.1)
-                yield make_chunk(content=block)
-            except asyncio.TimeoutError:
-                continue
-
-        # Drain any remaining
-        while not queue.empty():
-            block = await queue.get()
-            yield make_chunk(content=block)
-
-    finally:
-        try:
-            await task
-        except Exception as e:
-            yield make_chunk(error={'message': str(e), 'type': type(e).__name__})
-
-    # Final chunk: finish_reason
-    yield make_chunk(finish_reason='stop')
-    yield "data: [DONE]\n\n"
-
-
-def read_api_key(request: Request) -> str:
-    """
-    Extracts the Bearer token from the Authorization header.
-    returns '' if not present.
-    """
-    auth = request.headers.get("authorization")
-    if auth and auth.lower().startswith("bearer "):
-        return auth[7:].strip()
-    return ""
-
-
-def check_api_key(api_key: Optional[str]) -> Group:
-    for group_name, group in env.config.groups.items():
-        if api_key in group.api_keys:
-            return group_name
-
-
-async def chat_completions(request: ChatCompletionRequest, raw_request: Request) -> Response:
-    """
-    Endpoint for chat completions that mimics OpenAI's API structure.
-    Streams the response from the LLM using microcore.
-    """
-    if not env.config.enabled:
-        raise HTTPException(
-            status_code=503,
-            detail={
-                "error": {
-                    "message": "The service is disabled.",
-                    "type": "service_unavailable",
-                    "param": None,
-                    "code": "service_disabled",
-                }
-            },
-        )
-    api_key = read_api_key(raw_request)
-    group: str | bool | None = (env.config.check_api_key)(api_key)
-    if not group:
-        raise HTTPException(
-            status_code=403,
-            detail={
-                "error": {
-                    "message": "Incorrect API key provided: "
-                               "your API key is invalid, expired, or revoked.",
-                    "type": "invalid_request_error",
-                    "param": None,
-                    "code": "invalid_api_key",
-                }
-            },
-        )
-
-    llm_params = request.dict(exclude=['messages'], exclude_none=True)
-
-    connection, llm_params["model"] = resolve_connection_and_model(
-        env.config,
-        llm_params.get("model", "default_model")
-    )
-    logging.debug(
-        "Resolved routing for [%s] --> connection: %s, model: %s",
-        request.model,
-        connection,
-        llm_params["model"]
-    )
-
-    if not env.config.groups[group].allows_connecting_to(connection):
-        raise HTTPException(
-            status_code=403,
-            detail={
-                "error": {
-                    "message": f"Your API key does not allow using the '{connection}' connection.",
-                    "type": "invalid_request_error",
-                    "param": None,
-                    "code": "connection_not_allowed",
-                }
-            },
-        )
-
-    async_llm_func = env.connections[connection]
-
-    logging.info("Querying LLM... params: %s", llm_params)
-    if request.stream:
-        return StreamingResponse(
-            process_stream(async_llm_func, request.messages, llm_params),
-            media_type="text/event-stream"
-        )
-    out = await async_llm_func(request.messages, **llm_params)
-    logging.info("LLM response: %s", out)
-    return JSONResponse(
-        {
-            "choices": [
-                {
-                    "index": 0,
-                    "message": {"role": "assistant", "content": str(out)},
-                    "finish_reason": "stop"
-                }
-            ]
-        }
-    )
+import asyncio
+import fnmatch
+import json
+import logging
+import secrets
+import time
+from typing import List, Optional
+
+import microcore as mc
+from fastapi import HTTPException
+from pydantic import BaseModel
+from starlette.requests import Request
+from starlette.responses import JSONResponse, Response, StreamingResponse
+
+from .bootstrap import env
+from .config import Config, Group
+
+
+class ChatCompletionRequest(BaseModel):
+    model: str
+    messages: List[mc.Msg]
+    stream: Optional[bool] = None
+    max_tokens: Optional[int] = None
+    temperature: Optional[float] = None
+    top_p: Optional[float] = None
+    n: Optional[int] = None
+    stop: Optional[List[str]] = None
+    presence_penalty: Optional[float] = None
+    frequency_penalty: Optional[float] = None
+    user: Optional[str] = None
+
+
+def resolve_connection_and_model(config: Config, external_model: str) -> tuple[str, str]:
+    for model_match, rule in config.routing.items():
+        if fnmatch.fnmatchcase(external_model, model_match):
+            connection_name, model_part = rule.split(".", 1)
+            if connection_name not in config.connections:
+                raise ValueError(
+                    f"Routing selected unknown connection '{connection_name}'. "
+                    f"Defined connections: {', '.join(config.connections.keys()) or '(none)'}"
+                )
+
+            resolved_model = external_model if model_part == "*" else model_part
+            return connection_name, resolved_model
+
+    raise ValueError(
+        f"No routing rule matched model '{external_model}'. "
+        "Add a catch-all rule like \"*\" = \"openai.gpt-3.5-turbo\" if desired."
+    )
+
+
+async def process_stream(async_llm_func, prompt, llm_params):
+    queue = asyncio.Queue()
+    stream_id = f"chatcmpl-{secrets.token_hex(12)}"
+    created = int(time.time())
+
+    async def callback(chunk):
+        await queue.put(chunk)
+
+    def make_chunk(delta=None, content=None, finish_reason=None, error=None) -> str:
+        if delta is None:
+            delta = dict(content=str(content)) if content is not None else dict()
+        obj = {
+            "id": stream_id,
+            "object": "chat.completion.chunk",
+            "created": created,
+            "choices": [{"index": 0, "delta": delta}],
+        }
+        if error is not None:
+            obj['error'] = {'message': str(error), 'type': type(error).__name__}
+            if finish_reason is None:
+                finish_reason = 'error'
+        if finish_reason is not None:
+            obj['choices'][0]['finish_reason'] = finish_reason
+        return "data: " + json.dumps(obj) + "\n\n"
+
+    task = asyncio.create_task(
+        async_llm_func(prompt, **llm_params, callback=callback)
+    )
+
+    try:
+        # Initial chunk: role
+        yield make_chunk(delta={'role': 'assistant'})
+
+        while not task.done():
+            try:
+                block = await asyncio.wait_for(queue.get(), timeout=0.1)
+                yield make_chunk(content=block)
+            except asyncio.TimeoutError:
+                continue
+
+        # Drain any remaining
+        while not queue.empty():
+            block = await queue.get()
+            yield make_chunk(content=block)
+
+    finally:
+        try:
+            await task
+        except Exception as e:
+            yield make_chunk(error={'message': str(e), 'type': type(e).__name__})
+
+    # Final chunk: finish_reason
+    yield make_chunk(finish_reason='stop')
+    yield "data: [DONE]\n\n"
+
+
+def read_api_key(request: Request) -> str:
+    """
+    Extracts the Bearer token from the Authorization header.
+    returns '' if not present.
+    """
+    auth = request.headers.get("authorization")
+    if auth and auth.lower().startswith("bearer "):
+        return auth[7:].strip()
+    return ""
+
+
+def check_api_key(api_key: Optional[str]) -> Group:
+    for group_name, group in env.config.groups.items():
+        if api_key in group.api_keys:
+            return group_name
+
+
+async def chat_completions(request: ChatCompletionRequest, raw_request: Request) -> Response:
+    """
+    Endpoint for chat completions that mimics OpenAI's API structure.
+    Streams the response from the LLM using microcore.
+    """
+    if not env.config.enabled:
+        raise HTTPException(
+            status_code=503,
+            detail={
+                "error": {
+                    "message": "The service is disabled.",
+                    "type": "service_unavailable",
+                    "param": None,
+                    "code": "service_disabled",
+                }
+            },
+        )
+    api_key = read_api_key(raw_request)
+    group: str | bool | None = (env.config.check_api_key)(api_key)
+    if not group:
+        raise HTTPException(
+            status_code=403,
+            detail={
+                "error": {
+                    "message": "Incorrect API key provided: "
+                               "your API key is invalid, expired, or revoked.",
+                    "type": "invalid_request_error",
+                    "param": None,
+                    "code": "invalid_api_key",
+                }
+            },
+        )
+
+    llm_params = request.model_dump(exclude={'messages'}, exclude_none=True)
+
+    connection, llm_params["model"] = resolve_connection_and_model(
+        env.config,
+        llm_params.get("model", "default_model")
+    )
+    logging.debug(
+        "Resolved routing for [%s] --> connection: %s, model: %s",
+        request.model,
+        connection,
+        llm_params["model"]
+    )
+
+    if not env.config.groups[group].allows_connecting_to(connection):
+        raise HTTPException(
+            status_code=403,
+            detail={
+                "error": {
+                    "message": f"Your API key does not allow using the '{connection}' connection.",
+                    "type": "invalid_request_error",
+                    "param": None,
+                    "code": "connection_not_allowed",
+                }
+            },
+        )
+
+    async_llm_func = env.connections[connection]
+
+    logging.info("Querying LLM... params: %s", llm_params)
+    if request.stream:
+        return StreamingResponse(
+            process_stream(async_llm_func, request.messages, llm_params),
+            media_type="text/event-stream"
+        )
+    out = await async_llm_func(request.messages, **llm_params)
+    logging.info("LLM response: %s", out)
+    return JSONResponse(
+        {
+            "choices": [
+                {
+                    "index": 0,
+                    "message": {"role": "assistant", "content": str(out)},
+                    "finish_reason": "stop"
+                }
+            ]
+        }
+    )

{inference_proxy-0.2.0 → inference_proxy-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "inference-proxy"
-version = "0.2.0"
+version = "0.2.2"
 description = "\"Inference Proxy\" is OpenAI-compatible http proxy server for inferencing various LLMs capable of working with Google, Anthropic, OpenAI APIs, local PyTorch inference, etc."
 readme = "README.md"
 keywords = ["llm", "large language models", "ai", "gpt", "openai", "proxy", "http", "proxy-server"]
@@ -15,7 +15,7 @@ classifiers = [
     "License :: OSI Approved :: MIT License",
 ]
 dependencies = [
-    "ai-microcore~=4.3.2",
+    "ai-microcore~=4.4.3",
     "fastapi~=0.116.1",
     "uvicorn>=0.22.0",
     "typer>=0.16.1",
@@ -47,4 +47,3 @@ pytest = "^7.4.3"
 
 [tool.poetry.scripts]
 inference-proxy = "lm_proxy.app:cli_app"
-lm_proxy = "lm_proxy.app:cli_app"

inference_proxy-0.2.0/PKG-INFO

@@ -1,81 +0,0 @@
-Metadata-Version: 2.3
-Name: inference-proxy
-Version: 0.2.0
-Summary: "Inference Proxy" is OpenAI-compatible http proxy server for inferencing various LLMs capable of working with Google, Anthropic, OpenAI APIs, local PyTorch inference, etc.
-License: MIT License
-         
-         Copyright (c) 2025 Vitalii Stepanenko
-         
-         Permission is hereby granted, free of charge, to any person obtaining a copy
-         of this software and associated documentation files (the "Software"), to deal
-         in the Software without restriction, including without limitation the rights
-         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-         copies of the Software, and to permit persons to whom the Software is
-         furnished to do so, subject to the following conditions:
-         
-         The above copyright notice and this permission notice shall be included in all
-         copies or substantial portions of the Software.
-         
-         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-         SOFTWARE.
-Keywords: llm,large language models,ai,gpt,openai,proxy,http,proxy-server
-Author: Vitalii Stepanenko
-Author-email: mail@vitalii.in
-Maintainer: Vitalii Stepanenko
-Maintainer-email: mail@vitalii.in
-Requires-Python: >=3.10,<4
-Classifier: Intended Audience :: Developers
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: License :: OSI Approved :: MIT License
-Requires-Dist: ai-microcore (>=4.3.2,<4.4.0)
-Requires-Dist: fastapi (>=0.116.1,<0.117.0)
-Requires-Dist: typer (>=0.16.1)
-Requires-Dist: uvicorn (>=0.22.0)
-Project-URL: Source Code, https://github.com/Nayjest/lm-proxy
-Description-Content-Type: text/markdown
-
-<p align="right">
-<a href="https://pypi.org/project/lm-proxy/" target="_blank"><img src="https://badge.fury.io/py/lm-proxy.svg" alt="PYPI Release"></a>
-<a href="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml" target="_blank"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml/badge.svg" alt="Code Style"></a>
-<a href="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml" target="_blank"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
-<a href="https://github.com/Nayjest/lm-proxy/blob/main/LICENSE" target="_blank"><img src="https://img.shields.io/static/v1?label=license&message=MIT&color=d08aff" alt="License"></a>
-</p>
-
-# Inference Proxy
-
-**Inference Proxy** is OpenAI-compatible http proxy server for inferencing various LLMs capable of working with Google, Anthropic, OpenAI APIs, local PyTorch inference, etc.
-
-**Development Status**: bookmark it and go away, it is still in early development.
-
-## ✨ Features
-
-- @todo
-
-
-## 🚀 Quickstart
-```bash
-# Install Inference Proxy via pip
-pip install inference-proxy
-
-```
-
-## 🤝 Contributing
-
-We ❤️ contributions! See [CONTRIBUTING.md](CONTRIBUTING.md).
-
-## 📝 License
-
-Licensed under the [MIT License](LICENSE).
-
-© 2022&mdash;2025 [Vitalii Stepanenko](mailto:mail@vitaliy.in)
-

inference_proxy-0.2.0/README.md

@@ -1,34 +0,0 @@
-<p align="right">
-<a href="https://pypi.org/project/lm-proxy/" target="_blank"><img src="https://badge.fury.io/py/lm-proxy.svg" alt="PYPI Release"></a>
-<a href="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml" target="_blank"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml/badge.svg" alt="Code Style"></a>
-<a href="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml" target="_blank"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
-<a href="https://github.com/Nayjest/lm-proxy/blob/main/LICENSE" target="_blank"><img src="https://img.shields.io/static/v1?label=license&message=MIT&color=d08aff" alt="License"></a>
-</p>
-
-# Inference Proxy
-
-**Inference Proxy** is OpenAI-compatible http proxy server for inferencing various LLMs capable of working with Google, Anthropic, OpenAI APIs, local PyTorch inference, etc.
-
-**Development Status**: bookmark it and go away, it is still in early development.
-
-## ✨ Features
-
-- @todo
-
-
-## 🚀 Quickstart
-```bash
-# Install Inference Proxy via pip
-pip install inference-proxy
-
-```
-
-## 🤝 Contributing
-
-We ❤️ contributions! See [CONTRIBUTING.md](CONTRIBUTING.md).
-
-## 📝 License
-
-Licensed under the [MIT License](LICENSE).
-
-© 2022&mdash;2025 [Vitalii Stepanenko](mailto:mail@vitaliy.in)

inference_proxy-0.2.0/lm_proxy/bootstrap.py

@@ -1,70 +0,0 @@
-import sys
-import logging
-import inspect
-from datetime import datetime
-
-
-import microcore as mc
-from microcore import ui
-from microcore.configuration import get_bool_from_env
-from dotenv import load_dotenv
-
-from .config import Config
-
-
-def setup_logging(log_level: int = logging.INFO):
-    class CustomFormatter(logging.Formatter):
-        def format(self, record):
-            dt = datetime.fromtimestamp(record.created).strftime("%H:%M:%S")
-            message, level_name = record.getMessage(), record.levelname
-            if record.levelno == logging.WARNING:
-                message = mc.ui.yellow(message)
-                level_name = mc.ui.yellow(level_name)
-            if record.levelno >= logging.ERROR:
-                message = mc.ui.red(message)
-                level_name = mc.ui.red(level_name)
-            return f"{dt} {level_name}: {message}"
-
-    handler = logging.StreamHandler()
-    handler.setFormatter(CustomFormatter())
-    logging.basicConfig(level=log_level, handlers=[handler])
-
-
-class Env:
-    config: Config
-    connections: dict[str, mc.types.LLMAsyncFunctionType]
-    debug: bool
-
-
-env = Env()
-
-
-def bootstrap(config_file: str = 'config.toml'):
-    load_dotenv('.env', override=True)
-    env.debug = '--debug' in sys.argv or get_bool_from_env('LM_PROXY_DEBUG', False)
-    setup_logging(logging.DEBUG if env.debug else logging.INFO)
-    logging.info(
-        f"Bootstrapping {ui.yellow('lm_proxy')} "
-        f"using configuration: {ui.blue(config_file)} "
-        f"{'[DEBUG: ON]' if env.debug else ''}..."
-    )
-
-    env.config = Config.load(config_file)
-    env.connections = dict()
-
-    for conn_name, conn_config in env.config.connections.items():
-        logging.info(f"Initializing '{conn_name}' connection...")
-        try:
-            if inspect.iscoroutinefunction(conn_config):
-                env.connections[conn_name] = conn_config
-            else:
-                mc.configure(
-                    **conn_config,
-                    EMBEDDING_DB_TYPE=mc.EmbeddingDbType.NONE
-                )
-                env.connections[conn_name] = mc.env().llm_async_function
-        except mc.LLMConfigError as e:
-            raise ValueError(f"Error in configuration for connection '{conn_name}': {e}")
-
-    logging.info(f"Done initializing {len(env.connections)} connections.")
-    mc.logging.LoggingConfig.OUTPUT_METHOD = logging.info