llms-py 2.0.24__tar.gz → 2.0.25__tar.gz

{llms_py-2.0.24 → llms_py-2.0.25}/LICENSE

@@ -1,6 +1,5 @@
 Copyright (c) 2007-present, Demis Bellot, ServiceStack, Inc.
 https://servicestack.net
-All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
@@ -9,7 +8,7 @@ modification, are permitted provided that the following conditions are met:
     * Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.
-    * Neither the name of the ServiceStack nor the
+    * Neither the name of the copyright holder nor the
       names of its contributors may be used to endorse or promote products
       derived from this software without specific prior written permission.
 

{llms_py-2.0.24/llms_py.egg-info → llms_py-2.0.25}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llms-py
-Version: 2.0.24
+Version: 2.0.25
 Summary: A lightweight CLI tool and OpenAI-compatible server for querying multiple Large Language Model (LLM) providers
 Home-page: https://github.com/ServiceStack/llms
 Author: ServiceStack
@@ -50,7 +50,7 @@ Configure additional providers and models in [llms.json](llms/llms.json)
 
 ## Features
 
-- **Lightweight**: Single [llms.py](llms.py) Python file with single `aiohttp` dependency
+- **Lightweight**: Single [llms.py](https://github.com/ServiceStack/llms/blob/main/llms/main.py) Python file with single `aiohttp` dependency
 - **Multi-Provider Support**: OpenRouter, Ollama, Anthropic, Google, OpenAI, Grok, Groq, Qwen, Z.ai, Mistral
 - **OpenAI-Compatible API**: Works with any client that supports OpenAI's chat completion API
 - **Built-in Analytics**: Built-in analytics UI to visualize costs, requests, and token usage
@@ -68,24 +68,100 @@ Configure additional providers and models in [llms.json](llms/llms.json)
 
 Access all your local all remote LLMs with a single ChatGPT-like UI:
 
-[![](https://servicestack.net/img/posts/llms-py-ui/bg.webp)](https://servicestack.net/posts/llms-py-ui)
+[![](https://servicestack.net/img/posts/llms-py-ui/bg.webp?)](https://servicestack.net/posts/llms-py-ui)
 
 **Monthly Costs Analysis**
 
 [![](https://servicestack.net/img/posts/llms-py-ui/analytics-costs.webp)](https://servicestack.net/posts/llms-py-ui)
 
+**Monthly Token Usage**
+
+[![](https://servicestack.net/img/posts/llms-py-ui/analytics-tokens.webp)](https://servicestack.net/posts/llms-py-ui)
+
 **Monthly Activity Log**
 
 [![](https://servicestack.net/img/posts/llms-py-ui/analytics-activity.webp)](https://servicestack.net/posts/llms-py-ui)
 
 [More Features and Screenshots](https://servicestack.net/posts/llms-py-ui).
 
+**Check Provider Reliability and Response Times**
+
+Check the status of configured providers to test if they're configured correctly, reachable and what their response times is for the simplest `1+1=` request:
+
+```bash
+# Check all models for a provider:
+llms --check groq
+
+# Check specific models for a provider:
+llms --check groq kimi-k2 llama4:400b gpt-oss:120b
+```
+
+[![llms-check.webp](https://servicestack.net/img/posts/llms-py-ui/llms-check.webp)](https://servicestack.net/img/posts/llms-py-ui/llms-check.webp)
+
+As they're a good indicator for the reliability and speed you can expect from different providers we've created a 
+[test-providers.yml](https://github.com/ServiceStack/llms/actions/workflows/test-providers.yml) GitHub Action to
+test the response times for all configured providers and models, the results of which will be frequently published to
+[/checks/latest.txt](https://github.com/ServiceStack/llms/blob/main/docs/checks/latest.txt)
+
 ## Installation
 
+### Using pip
+
 ```bash
 pip install llms-py
 ```
 
+### Using Docker
+
+**a) Simple - Run in a Docker container:**
+
+Run the server on port `8000`:
+
+```bash
+docker run -p 8000:8000 -e GROQ_API_KEY=$GROQ_API_KEY ghcr.io/servicestack/llms:latest
+```
+
+Get the latest version:
+
+```bash
+docker pull ghcr.io/servicestack/llms:latest
+```
+
+Use custom `llms.json` and `ui.json` config files outside of the container (auto created if they don't exist):
+
+```bash
+docker run -p 8000:8000 -e GROQ_API_KEY=$GROQ_API_KEY \
+  -v ~/.llms:/home/llms/.llms \
+  ghcr.io/servicestack/llms:latest
+```
+
+**b) Recommended - Use Docker Compose:**
+
+Download and use [docker-compose.yml](https://raw.githubusercontent.com/ServiceStack/llms/refs/heads/main/docker-compose.yml):
+
+```bash
+curl -O https://raw.githubusercontent.com/ServiceStack/llms/refs/heads/main/docker-compose.yml
+```
+
+Update API Keys in `docker-compose.yml` then start the server:
+
+```bash
+docker-compose up -d
+```
+
+**c) Build and run local Docker image from source:**
+
+```bash
+git clone https://github.com/ServiceStack/llms
+
+docker-compose -f docker-compose.local.yml up -d --build
+```
+
+After the container starts, you can access the UI and API at `http://localhost:8000`.
+
+
+See [DOCKER.md](DOCKER.md) for detailed instructions on customizing configuration files.
+
 ## Quick Start
 
 ### 1. Set API Keys
@@ -112,34 +188,42 @@ export OPENROUTER_API_KEY="..."
 | z.ai            | `ZAI_API_KEY`             | Z.ai API key        | `sk-...` |
 | mistral         | `MISTRAL_API_KEY`         | Mistral API key     | `...` |
 
-### 2. Enable Providers
+### 2. Run Server
 
-Enable the providers you want to use:
+Start the UI and an OpenAI compatible API on port **8000**:
 
 ```bash
-# Enable providers with free models and free tiers
-llms --enable openrouter_free google_free groq
-
-# Enable paid providers
-llms --enable openrouter anthropic google openai mistral grok qwen
+llms --serve 8000
 ```
 
-### 3. Run UI
+Launches UI at `http://localhost:8000` and OpenAI Endpoint at `http://localhost:8000/v1/chat/completions`.
 
-Start the UI and an OpenAI compatible API on port **8000**:
+To see detailed request/response logging, add `--verbose`:
 
 ```bash
-llms --serve 8000
+llms --serve 8000 --verbose
 ```
 
-Launches the UI at `http://localhost:8000` and an OpenAI Endpoint at `http://localhost:8000/v1/chat/completions`.
-
-### 4. Use llms.py CLI
+### Use llms.py CLI
 
 ```bash
 llms "What is the capital of France?"
 ```
 
+### Enable Providers
+
+Any providers that have their API Keys set and enabled in `llms.json` are automatically made available.
+
+Providers can be enabled or disabled in the UI at runtime next to the model selector, or on the command line:
+
+```bash
+# Disable free providers with free models and free tiers
+llms --disable openrouter_free codestral google_free groq
+
+# Enable paid providers
+llms --enable openrouter anthropic google openai grok z.ai qwen mistral
+```
+
 ## Configuration
 
 The configuration file [llms.json](llms/llms.json) is saved to `~/.llms/llms.json` and defines available providers, models, and default settings. Key sections:
@@ -147,6 +231,10 @@ The configuration file [llms.json](llms/llms.json) is saved to `~/.llms/llms.jso
 ### Defaults
 - `headers`: Common HTTP headers for all requests
 - `text`: Default chat completion request template for text prompts
+- `image`: Default chat completion request template for image prompts
+- `audio`: Default chat completion request template for audio prompts
+- `file`: Default chat completion request template for file prompts
+- `check`: Check request template for testing provider connectivity
 
 ### Providers
 
@@ -156,7 +244,9 @@ Each provider configuration includes:
 - `api_key`: API key (supports environment variables with `$VAR_NAME`)
 - `base_url`: API endpoint URL
 - `models`: Model name mappings (local name → provider name)
-
+- `pricing`: Pricing per token (input/output) for each model
+- `default_pricing`: Default pricing if not specified in `pricing`
+- `check`: Check request template for testing provider connectivity
 
 ## Command Line Usage
 
@@ -498,9 +588,6 @@ llms --verbose --logprefix "[DEBUG] " "Hello world"
 # Set default model (updates config file)
 llms --default grok-4
 
-# Update llms.py to latest version
-llms --update
-
 # Pass custom parameters to chat request (URL-encoded)
 llms --args "temperature=0.7&seed=111" "What is 2+2?"
 
@@ -570,19 +657,10 @@ When you set a default model:
 
 ### Updating llms.py
 
-The `--update` option downloads and installs the latest version of `llms.py` from the GitHub repository:
-
 ```bash
-# Update to latest version
-llms --update
+pip install llms-py --upgrade
 ```
 
-This command:
-- Downloads the latest `llms.py` from `github.com/ServiceStack/llms/blob/main/llms/main.py`
-- Overwrites your current `llms.py` file with the latest version
-- Preserves your existing configuration file (`llms.json`)
-- Requires an internet connection to download the update
-
 ### Beautiful rendered Markdown
 
 Pipe Markdown output to [glow](https://github.com/charmbracelet/glow) to beautifully render it in the terminal:
@@ -818,35 +896,249 @@ Example: If both OpenAI and OpenRouter support `kimi-k2`, the request will first
 
 ## Usage
 
-    Run `llms` without arguments to see the help screen:
-
-    usage: llms.py [-h] [--config FILE] [-m MODEL] [--chat REQUEST] [-s PROMPT] [--image IMAGE] [--audio AUDIO]
-                  [--file FILE] [--raw] [--list] [--serve PORT] [--enable PROVIDER] [--disable PROVIDER]
-                  [--default MODEL] [--init] [--logprefix PREFIX] [--verbose] [--update]
+    usage: llms [-h] [--config FILE] [-m MODEL] [--chat REQUEST] [-s PROMPT] [--image IMAGE] [--audio AUDIO] [--file FILE]
+                [--args PARAMS] [--raw] [--list] [--check PROVIDER] [--serve PORT] [--enable PROVIDER] [--disable PROVIDER]
+                [--default MODEL] [--init] [--root PATH] [--logprefix PREFIX] [--verbose]
 
-    llms
+    llms v2.0.24
 
     options:
       -h, --help            show this help message and exit
       --config FILE         Path to config file
-      -m MODEL, --model MODEL
-                            Model to use
+      -m, --model MODEL     Model to use
       --chat REQUEST        OpenAI Chat Completion Request to send
-      -s PROMPT, --system PROMPT
-                            System prompt to use for chat completion
+      -s, --system PROMPT   System prompt to use for chat completion
       --image IMAGE         Image input to use in chat completion
       --audio AUDIO         Audio input to use in chat completion
       --file FILE           File input to use in chat completion
+      --args PARAMS         URL-encoded parameters to add to chat request (e.g. "temperature=0.7&seed=111")
       --raw                 Return raw AI JSON response
       --list                Show list of enabled providers and their models (alias ls provider?)
+      --check PROVIDER      Check validity of models for a provider
       --serve PORT          Port to start an OpenAI Chat compatible server on
       --enable PROVIDER     Enable a provider
       --disable PROVIDER    Disable a provider
       --default MODEL       Configure the default model to use
       --init                Create a default llms.json
+      --root PATH           Change root directory for UI files
       --logprefix PREFIX    Prefix used in log messages
       --verbose             Verbose output
-      --update              Update to latest version
+
+## Docker Deployment
+
+### Quick Start with Docker
+
+The easiest way to run llms-py is using Docker:
+
+```bash
+# Using docker-compose (recommended)
+docker-compose up -d
+
+# Or pull and run directly
+docker run -p 8000:8000 \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest
+```
+
+### Docker Images
+
+Pre-built Docker images are automatically published to GitHub Container Registry:
+
+- **Latest stable**: `ghcr.io/servicestack/llms:latest`
+- **Specific version**: `ghcr.io/servicestack/llms:v2.0.24`
+- **Main branch**: `ghcr.io/servicestack/llms:main`
+
+### Environment Variables
+
+Pass API keys as environment variables:
+
+```bash
+docker run -p 8000:8000 \
+  -e OPENROUTER_API_KEY="sk-or-..." \
+  -e GROQ_API_KEY="gsk_..." \
+  -e GOOGLE_FREE_API_KEY="AIza..." \
+  -e ANTHROPIC_API_KEY="sk-ant-..." \
+  -e OPENAI_API_KEY="sk-..." \
+  ghcr.io/servicestack/llms:latest
+```
+
+### Using docker-compose
+
+Create a `docker-compose.yml` file (or use the one in the repository):
+
+```yaml
+version: '3.8'
+
+services:
+  llms:
+    image: ghcr.io/servicestack/llms:latest
+    ports:
+      - "8000:8000"
+    environment:
+      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
+      - GROQ_API_KEY=${GROQ_API_KEY}
+      - GOOGLE_FREE_API_KEY=${GOOGLE_FREE_API_KEY}
+    volumes:
+      - llms-data:/home/llms/.llms
+    restart: unless-stopped
+
+volumes:
+  llms-data:
+```
+
+Create a `.env` file with your API keys:
+
+```bash
+OPENROUTER_API_KEY=sk-or-...
+GROQ_API_KEY=gsk_...
+GOOGLE_FREE_API_KEY=AIza...
+```
+
+Start the service:
+
+```bash
+docker-compose up -d
+```
+
+### Building Locally
+
+Build the Docker image from source:
+
+```bash
+# Using the build script
+./docker-build.sh
+
+# Or manually
+docker build -t llms-py:latest .
+
+# Run your local build
+docker run -p 8000:8000 \
+  -e OPENROUTER_API_KEY="your-key" \
+  llms-py:latest
+```
+
+### Volume Mounting
+
+To persist configuration and analytics data between container restarts:
+
+```bash
+# Using a named volume (recommended)
+docker run -p 8000:8000 \
+  -v llms-data:/home/llms/.llms \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest
+
+# Or mount a local directory
+docker run -p 8000:8000 \
+  -v $(pwd)/llms-config:/home/llms/.llms \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest
+```
+
+### Custom Configuration Files
+
+Customize llms-py behavior by providing your own `llms.json` and `ui.json` files:
+
+**Option 1: Mount a directory with custom configs**
+
+```bash
+# Create config directory with your custom files
+mkdir -p config
+# Add your custom llms.json and ui.json to config/
+
+# Mount the directory
+docker run -p 8000:8000 \
+  -v $(pwd)/config:/home/llms/.llms \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest
+```
+
+**Option 2: Mount individual config files**
+
+```bash
+docker run -p 8000:8000 \
+  -v $(pwd)/my-llms.json:/home/llms/.llms/llms.json:ro \
+  -v $(pwd)/my-ui.json:/home/llms/.llms/ui.json:ro \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest
+```
+
+**With docker-compose:**
+
+```yaml
+volumes:
+  # Use local directory
+  - ./config:/home/llms/.llms
+
+  # Or mount individual files
+  # - ./my-llms.json:/home/llms/.llms/llms.json:ro
+  # - ./my-ui.json:/home/llms/.llms/ui.json:ro
+```
+
+The container will auto-create default config files on first run if they don't exist. You can customize these to:
+- Enable/disable specific providers
+- Add or remove models
+- Configure API endpoints
+- Set custom pricing
+- Customize chat templates
+- Configure UI settings
+
+See [DOCKER.md](DOCKER.md) for detailed configuration examples.
+
+### Custom Port
+
+Change the port mapping to run on a different port:
+
+```bash
+# Run on port 3000 instead of 8000
+docker run -p 3000:8000 \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest
+```
+
+### Docker CLI Usage
+
+You can also use the Docker container for CLI commands:
+
+```bash
+# Run a single query
+docker run --rm \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest \
+  llms "What is the capital of France?"
+
+# List available models
+docker run --rm \
+  -e OPENROUTER_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest \
+  llms --list
+
+# Check provider status
+docker run --rm \
+  -e GROQ_API_KEY="your-key" \
+  ghcr.io/servicestack/llms:latest \
+  llms --check groq
+```
+
+### Health Checks
+
+The Docker image includes a health check that verifies the server is responding:
+
+```bash
+# Check container health
+docker ps
+
+# View health check logs
+docker inspect --format='{{json .State.Health}}' llms-server
+```
+
+### Multi-Architecture Support
+
+The Docker images support multiple architectures:
+- `linux/amd64` (x86_64)
+- `linux/arm64` (ARM64/Apple Silicon)
+
+Docker will automatically pull the correct image for your platform.
 
 ## Troubleshooting
 
@@ -908,9 +1200,10 @@ This shows:
 
 ### Project Structure
 
-- `llms.py` - Main script with CLI and server functionality
-- `llms.json` - Default configuration file
-- `requirements.txt` - Python dependencies
+- `llms/main.py` - Main script with CLI and server functionality
+- `llms/llms.json` - Default configuration file
+- `llms/ui.json` - UI configuration file
+- `requirements.txt` - Python dependencies (aiohttp)
 
 ### Provider Classes