local-ai-runtime 0.1.0__tar.gz

local_ai_runtime-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Elie Khalil
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

local_ai_runtime-0.1.0/PKG-INFO

@@ -0,0 +1,29 @@
+Metadata-Version: 2.4
+Name: local-ai-runtime
+Version: 0.1.0
+Summary: Local AI chat runtime — BYOK hybrid inference (local GGUF, ollama, vLLM, OpenAI, Anthropic, Gemini, any OpenAI-compatible)
+Requires-Python: >=3.14
+License-File: LICENSE
+Requires-Dist: fastapi>=0.100.0
+Requires-Dist: uvicorn>=0.20.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: httpx>=0.25.0
+Provides-Extra: llama-cpp
+Requires-Dist: llama-cpp-python>=0.2.0; extra == "llama-cpp"
+Provides-Extra: ollama
+Provides-Extra: vllm
+Provides-Extra: openai
+Requires-Dist: openai>=1.0.0; extra == "openai"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.20.0; extra == "anthropic"
+Provides-Extra: gemini
+Requires-Dist: google-generativeai>=0.3.0; extra == "gemini"
+Provides-Extra: hf
+Requires-Dist: huggingface-hub>=0.20.0; extra == "hf"
+Provides-Extra: all
+Requires-Dist: llama-cpp-python>=0.2.0; extra == "all"
+Requires-Dist: openai>=1.0.0; extra == "all"
+Requires-Dist: anthropic>=0.20.0; extra == "all"
+Requires-Dist: google-generativeai>=0.3.0; extra == "all"
+Requires-Dist: huggingface-hub>=0.20.0; extra == "all"
+Dynamic: license-file

local_ai_runtime-0.1.0/README.md

@@ -0,0 +1,214 @@
+# local-ai-runtime
+
+> BYOK hybrid local AI chat runtime — run any model from anywhere.
+> Local GGUF (llama.cpp), Ollama, vLLM, OpenAI, Anthropic, Gemini, or any OpenAI-compatible endpoint.
+
+---
+
+## Quick Start
+
+### macOS (recommended via Homebrew)
+
+```bash
+brew install local-ai-runtime
+local-ai-runtime
+```
+
+### Any platform (uvx)
+
+```bash
+uvx local-ai-runtime
+```
+
+On macOS, you'll be prompted to use Homebrew instead for better native performance.
+
+### Development
+
+```bash
+git clone https://github.com/youruser/local-ai-runtime.git
+cd local-ai-runtime
+
+# Install dependencies with uv
+uv sync
+
+# Start the backend
+uv run local-ai-runtime
+
+# Start the frontend (separate terminal)
+cd frontend && npm install && npm run dev
+```
+
+---
+
+## What This Is
+
+A self-hosted AI chat application that supports **any inference backend** via BYOK (Bring Your Own Key):
+
+| Backend | Type | Config |
+|---------|------|--------|
+| **llama-cpp** | Local GGUF | `model_file` in config |
+| **Ollama** | Local | `ollama` running locally |
+| **vLLM** | Local | `vllm serve` running |
+| **OpenAI** | API (paid) | `OPENAI_API_KEY` env var |
+| **Anthropic** | API (paid) | `ANTHROPIC_API_KEY` env var |
+| **Gemini** | API (free tier) | `GEMINI_API_KEY` env var |
+| **OpenAI-compatible** | API (any) | Any LiteLLM/LM Studio/etc. |
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────┐
+│                   FRONTEND                      │
+│        Vite + React (dark theme chat UI)        │
+└────────────────────┬────────────────────────────┘
+                     │  HTTP + SSE
+                     ▼
+┌─────────────────────────────────────────────────┐
+│                 BACKEND API                     │
+│              FastAPI (Python)                   │
+│  Routes: /chat /chat/stream /models /config     │
+└────────────────────┬────────────────────────────┘
+                     │
+                     ▼
+┌─────────────────────────────────────────────────┐
+│              BYOK BACKEND SYSTEM                │
+│  ┌───────────┐ ┌─────────┐ ┌─────────────────┐ │
+│  │ llama-cpp │ │ ollama  │ │   openai/anthropic│
+│  └───────────┘ └─────────┘ └─────────────────┘ │
+│  ┌─────────┐ ┌───────┐ ┌────────────────────┐  │
+│  │  vLLM   │ │ gemini│ │ openai_compatible  │  │
+│  └─────────┘ └───────┘ └────────────────────┘  │
+└─────────────────────────────────────────────────┘
+```
+
+---
+
+## Configuration
+
+All configuration lives in `config/`. **Zero hardcoded values.**
+
+### server.config.json
+
+```json
+{
+  "host": "0.0.0.0",
+  "port": 8000,
+  "log_level": "info",
+  "cors_origins": ["http://localhost:5173"],
+  "models_dir": "./models"
+}
+```
+
+### model.config.json
+
+```json
+{
+  "backend_type": "llama-cpp",
+  "model_file": "mistral-7b-instruct.Q4_K_M.gguf",
+  "chat_template": "auto",
+  "context_length": 4096,
+  "system_prompt": "You are a helpful assistant.",
+  "generation": {
+    "temperature": 0.7,
+    "max_new_tokens": 512
+  }
+}
+```
+
+Set `backend_type` to any supported backend. For API backends, configure in `api_backends` section and set the corresponding API key environment variable.
+
+---
+
+## Model Detection
+
+Models are auto-detected from GGUF metadata:
+
+- Architecture (llama, mistral, phi, qwen, etc.)
+- Chat template (auto-mapped from architecture)
+- Context length
+- Quantization level
+- Parameter count
+
+No manual configuration needed — drop a `.gguf` file in `models/` and it works.
+
+---
+
+## Chat Templates
+
+Chat templates are auto-detected from:
+
+1. **Model metadata** — GGUF files contain architecture info that maps to templates
+2. **Conversation patterns** — the system detects `[INST]`, `<|im_start|>`, etc. from messages
+
+Supported templates: `chatml`, `llama3`, `llama2`, `mistral`, `alpaca`, `raw`, or `auto`.
+
+---
+
+## API Endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/status` | Health check, model state |
+| `GET` | `/models` | List available model files |
+| `GET` | `/models/detect` | Auto-detect model metadata |
+| `GET` | `/config` | Get active configuration |
+| `PUT` | `/config` | Update configuration |
+| `GET` | `/config/backends` | List available backends |
+| `POST` | `/chat` | Send message, get response |
+| `POST` | `/chat/stream` | Stream response (SSE) |
+| `GET` | `/conversations` | List conversations |
+| `POST` | `/conversations` | Create conversation |
+| `GET` | `/metrics` | Performance metrics |
+
+---
+
+## Directory Structure
+
+```
+local-ai-runtime/
+├── local_ai_runtime/          # New CLI + config system
+│   ├── cli.py                 # Entry point (uvx local-ai-runtime)
+│   ├── config.py              # Config loader
+│   └── server.py              # FastAPI app factory
+├── backend/                   # API routes + services
+│   ├── routes/                # FastAPI routers
+│   ├── services/              # Business logic
+│   ├── controllers/           # Request handlers
+│   └── models/                # Data classes
+├── model_runtime/             # Inference layer
+│   ├── backends/              # BYOK adapter system
+│   │   ├── base.py            # Abstract Backend interface
+│   │   ├── registry.py        # Dynamic backend loading
+│   │   ├── llama_cpp.py       # llama.cpp backend
+│   │   ├── ollama.py          # Ollama backend
+│   │   ├── vllm.py            # vLLM backend
+│   │   ├── openai_backend.py  # OpenAI API
+│   │   ├── anthropic_backend.py # Anthropic API
+│   │   ├── gemini_backend.py  # Google Gemini
+│   │   └── openai_compatible.py # Any OpenAI-compatible
+│   ├── detector.py            # GGUF metadata detection
+│   ├── chat_template_auto.py  # Auto chat template
+│   ├── prompt_formatter.py    # Template formatting
+│   └── inference_engine.py    # Legacy engine wrapper
+├── frontend/                  # Vite + React UI
+│   ├── app.jsx                # Root component
+│   ├── components/            # ChatWindow, InputBar, ModelSelector
+│   └── styles/                # Dark theme CSS
+├── config/                    # JSON config files
+├── models/                    # GGUF files (gitignored)
+├── huggingface/               # HF download support
+├── pyproject.toml             # uv project config
+└── .python-version            # Python 3.12
+```
+
+---
+
+## Requirements
+
+- Python 3.11+
+- Node.js 18+ (for frontend)
+- uv (package manager)
+- For local inference: llama-cpp-python, ollama, or vllm
+- For API inference: API keys as environment variables

local_ai_runtime-0.1.0/pyproject.toml

@@ -0,0 +1,40 @@
+[project]
+name = "local-ai-runtime"
+version = "0.1.0"
+description = "Local AI chat runtime — BYOK hybrid inference (local GGUF, ollama, vLLM, OpenAI, Anthropic, Gemini, any OpenAI-compatible)"
+requires-python = ">=3.14"
+dependencies = [
+    "fastapi>=0.100.0",
+    "uvicorn>=0.20.0",
+    "pydantic>=2.0.0",
+    "httpx>=0.25.0",
+]
+
+[project.optional-dependencies]
+llama-cpp = ["llama-cpp-python>=0.2.0"]
+ollama = []
+vllm = []
+openai = ["openai>=1.0.0"]
+anthropic = ["anthropic>=0.20.0"]
+gemini = ["google-generativeai>=0.3.0"]
+hf = ["huggingface-hub>=0.20.0"]
+all = [
+    "llama-cpp-python>=0.2.0",
+    "openai>=1.0.0",
+    "anthropic>=0.20.0",
+    "google-generativeai>=0.3.0",
+    "huggingface-hub>=0.20.0",
+]
+
+[project.scripts]
+local-ai-runtime = "src.local_ai_runtime.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["local_ai_runtime*", "model_runtime*", "backend*"]
+
+[dependency-groups]
+dev = [
+    "ruff>=0.1.0",
+    "pytest>=7.0.0",
+]

local_ai_runtime-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

local_ai_runtime-0.1.0/src/backend/controllers/chat_controller.py

@@ -0,0 +1,42 @@
+"""Controller for chat request handling."""
+
+from fastapi import HTTPException
+
+from services.inference_service import generate_response, InferenceError
+from models.chat_message import ChatMessage, Role
+from services.conversation_service import get_conversation
+
+
+async def handle_chat_request(request) -> dict:
+    user_message = ChatMessage(role=Role.USER, content=request.message)
+
+    conversation_history = None
+    if request.conversation_id:
+        conv = get_conversation(request.conversation_id)
+        if conv:
+            conversation_history = [
+                ChatMessage(role=Role(m["role"]), content=m["content"])
+                for m in conv["messages"]
+            ]
+
+    try:
+        result = await generate_response(
+            message=user_message,
+            conversation_id=request.conversation_id,
+            conversation_history=conversation_history,
+        )
+    except InferenceError as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+    # Auto-save user message and assistant response to conversation
+    if request.conversation_id:
+        from services.conversation_service import add_message_to_conversation
+        add_message_to_conversation(request.conversation_id, "user", request.message)
+        add_message_to_conversation(request.conversation_id, "assistant", result.content)
+
+    return {
+        "response": result.content,
+        "model": result.model_name,
+        "tokens_used": result.tokens_used,
+        "finish_reason": result.finish_reason,
+    }

local_ai_runtime-0.1.0/src/backend/controllers/model_controller.py

@@ -0,0 +1,15 @@
+"""Controller for model management request handling."""
+
+from services.model_service import scan_model_directory, get_active_config
+
+
+def list_available_models() -> dict:
+    models = scan_model_directory()
+    return {"models": models}
+
+
+def get_active_model() -> dict:
+    config = get_active_config()
+    if config is None:
+        return {"active_model": None}
+    return config.to_dict()

local_ai_runtime-0.1.0/src/backend/models/__init__.py

@@ -0,0 +1,2 @@
+from models.chat_message import ChatMessage, Conversation, Role
+from models.model_config import ModelConfig, GenerationParams, CHAT_TEMPLATES

local_ai_runtime-0.1.0/src/backend/models/chat_message.py

@@ -0,0 +1,80 @@
+# backend/models/chat_message.py
+#
+# Data model for a single chat message.
+#
+# STATUS: DEFINED — This can be used now even before inference works.
+# It's a plain data class with no dependencies.
+#
+# Usage:
+#   from models.chat_message import ChatMessage, Role
+#   msg = ChatMessage(role=Role.USER, content="Hello!")
+
+from enum import Enum
+from datetime import datetime, timezone
+
+
+class Role(str, Enum):
+    """
+    The role of the entity producing a message.
+    Mirrors the convention used in most chat template formats.
+    """
+    SYSTEM = "system"
+    USER = "user"
+    ASSISTANT = "assistant"
+
+    # Future: tool call results
+    # TOOL = "tool"
+
+
+class ChatMessage:
+    """
+    Represents a single message in a conversation.
+
+    Attributes:
+        role      (Role)     — Who produced this message
+        content   (str)      — The text content
+        timestamp (datetime) — When this message was created (UTC)
+        id        (str|None) — Optional unique ID for this message
+    """
+
+    def __init__(self, role: Role, content: str, timestamp: datetime | None = None, id: str | None = None):
+        self.role = role
+        self.content = content
+        self.timestamp = timestamp or datetime.now(timezone.utc)
+        self.id = id
+
+    def to_dict(self) -> dict:
+        return {
+            "role": self.role.value,
+            "content": self.content,
+            "timestamp": self.timestamp.isoformat(),
+            "id": self.id,
+        }
+
+    def __repr__(self) -> str:
+        preview = self.content[:60] + "..." if len(self.content) > 60 else self.content
+        return f"ChatMessage(role={self.role.value}, content='{preview}')"
+
+
+class Conversation:
+    """
+    A list of ChatMessages forming a conversation thread.
+
+    TODO (Phase 4.1): Add persistence, conversation ID, title generation.
+    """
+
+    def __init__(self, id: str | None = None):
+        self.id = id
+        self.messages: list[ChatMessage] = []
+
+    def add_message(self, message: ChatMessage) -> None:
+        self.messages.append(message)
+
+    def get_messages(self) -> list[ChatMessage]:
+        return self.messages
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "messages": [m.to_dict() for m in self.messages],
+        }

local_ai_runtime-0.1.0/src/backend/models/model_config.py

@@ -0,0 +1,145 @@
+"""Data models for model configuration and inference I/O."""
+
+from dataclasses import dataclass, field
+from typing import Optional
+
+
+CHAT_TEMPLATES = {
+    "auto",
+    "chatml",
+    "llama3",
+    "llama2",
+    "mistral",
+    "alpaca",
+    "raw",
+}
+
+BACKEND_TYPES = {
+    "llama-cpp",
+    "ollama",
+    "vllm",
+    "openai",
+    "anthropic",
+    "gemini",
+    "openai_compatible",
+}
+
+
+@dataclass
+class GenerationParams:
+    """Parameters controlling text generation."""
+    temperature: float = 0.7
+    top_p: float = 0.9
+    top_k: int = 40
+    repeat_penalty: float = 1.1
+    max_new_tokens: int = 512
+    min_new_tokens: int = 1
+    seed: int = -1
+
+    def to_dict(self) -> dict:
+        return {
+            "temperature": self.temperature,
+            "top_p": self.top_p,
+            "top_k": self.top_k,
+            "repeat_penalty": self.repeat_penalty,
+            "max_new_tokens": self.max_new_tokens,
+            "min_new_tokens": self.min_new_tokens,
+            "seed": self.seed,
+        }
+
+
+@dataclass
+class ModelConfig:
+    """Complete configuration for one model — all from config, no hardcoded defaults."""
+    model_file: str = ""
+    backend_type: str = "llama-cpp"
+    chat_template: str = "auto"
+    context_length: int = 4096
+    system_prompt: str = "You are a helpful assistant."
+    n_gpu_layers: int = 0
+    generation: GenerationParams = field(default_factory=GenerationParams)
+    profile_name: Optional[str] = None
+    models_dir: str = "./models"
+    # API backend overrides
+    openai_model: str = ""
+    anthropic_model: str = ""
+    gemini_model: str = ""
+    ollama_model: str = ""
+    vllm_model: str = ""
+    compatible_model: str = ""
+
+    def to_dict(self) -> dict:
+        return {
+            "profile_name": self.profile_name,
+            "backend_type": self.backend_type,
+            "model_file": self.model_file,
+            "chat_template": self.chat_template,
+            "context_length": self.context_length,
+            "system_prompt": self.system_prompt,
+            "n_gpu_layers": self.n_gpu_layers,
+            "models_dir": self.models_dir,
+            "generation": self.generation.to_dict(),
+            "openai_model": self.openai_model,
+            "anthropic_model": self.anthropic_model,
+            "gemini_model": self.gemini_model,
+            "ollama_model": self.ollama_model,
+            "vllm_model": self.vllm_model,
+            "compatible_model": self.compatible_model,
+        }
+
+    @classmethod
+    def from_dict(cls, data: dict) -> "ModelConfig":
+        gen_data = data.get("generation", {})
+        generation = GenerationParams(
+            temperature=gen_data.get("temperature", 0.7),
+            top_p=gen_data.get("top_p", 0.9),
+            top_k=gen_data.get("top_k", 40),
+            repeat_penalty=gen_data.get("repeat_penalty", 1.1),
+            max_new_tokens=gen_data.get("max_new_tokens", 512),
+            seed=gen_data.get("seed", -1),
+        )
+        return cls(
+            model_file=data.get("model_file", ""),
+            backend_type=data.get("backend_type", "llama-cpp"),
+            chat_template=data.get("chat_template", "auto"),
+            context_length=data.get("context_length", 4096),
+            system_prompt=data.get("system_prompt", "You are a helpful assistant."),
+            n_gpu_layers=data.get("n_gpu_layers", 0),
+            generation=generation,
+            profile_name=data.get("profile_name"),
+            models_dir=data.get("models_dir", "./models"),
+            openai_model=data.get("openai_model", ""),
+            anthropic_model=data.get("anthropic_model", ""),
+            gemini_model=data.get("gemini_model", ""),
+            ollama_model=data.get("ollama_model", ""),
+            vllm_model=data.get("vllm_model", ""),
+            compatible_model=data.get("compatible_model", ""),
+        )
+
+
+@dataclass
+class InferenceRequest:
+    """Internal representation of an inference request."""
+    prompt: str
+    system_prompt: str = ""
+    max_tokens: int = 512
+    temperature: float = 0.7
+    top_p: float = 0.9
+    top_k: int = 40
+    repeat_penalty: float = 1.1
+    seed: int = -1
+    stop_tokens: list[str] = field(default_factory=list)
+
+
+@dataclass
+class InferenceResponse:
+    """Internal representation of an inference response."""
+    text: str
+    tokens_prompt: int = 0
+    tokens_generated: int = 0
+    finish_reason: str = "stop"
+    model_name: str = ""
+
+    @property
+    def tokens_used(self) -> int:
+        return self.tokens_prompt + self.tokens_generated