llmjs2 1.1.1 → 1.3.0

package/docs/README.md

@@ -0,0 +1,47 @@
+# llmjs2 Documentation
+
+Welcome to the llmjs2 documentation! This folder contains all the documentation for using and understanding the llmjs2 library.
+
+## Documentation Overview
+
+### 📖 Getting Started
+
+- **[GET_STARTED.md](GET_STARTED.md)** - Quick setup guide for new users (5 minutes to first completion)
+
+### 🔧 Usage Guides
+
+- **[BASIC_USAGE.md](BASIC_USAGE.md)** - Core API patterns, configuration, and common use cases
+- **[ROUTER_GUIDE.md](ROUTER_GUIDE.md)** - Model routing and load balancing
+- **[GUARDRAILS_GUIDE.md](GUARDRAILS_GUIDE.md)** - Content filtering and request processing
+- **[SERVER_MODE.md](SERVER_MODE.md)** - Run llmjs2 as an OpenAI-compatible API server with routing
+- **[CLI.md](CLI.md)** - Command-line interface for server management
+
+## Quick Navigation
+
+### New to llmjs2?
+
+Start with **[GET_STARTED.md](GET_STARTED.md)** to get up and running quickly.
+
+### Want to use the API directly?
+
+Check out **[BASIC_USAGE.md](BASIC_USAGE.md)** for different API patterns and examples.
+
+### Need routing and load balancing?
+
+See **[ROUTER_GUIDE.md](ROUTER_GUIDE.md)** for intelligent model routing.
+
+### Need content filtering or custom processing?
+
+See **[GUARDRAILS_GUIDE.md](GUARDRAILS_GUIDE.md)** for guardrails and request processing.
+
+### Need to set up a server?
+
+See **[SERVER_MODE.md](SERVER_MODE.md)** for OpenAI-compatible server setup.
+
+### Prefer command-line tools?
+
+**[CLI.md](CLI.md)** covers the command-line interface and configuration files.
+
+## Contributing
+
+Documentation improvements are welcome! Please ensure any changes maintain consistency across all documentation files.

package/docs/ROUTER_GUIDE.md

@@ -0,0 +1,397 @@
+# llmjs2 Router Usage Guide
+
+The llmjs2 router provides intelligent model routing and load balancing capabilities, allowing you to distribute requests across multiple model deployments with different strategies.
+
+## Overview
+
+The router system enables:
+
+- **Load balancing** across models with the same name
+- **Multiple routing strategies** (default, random, sequential)
+- **Provider-agnostic routing** with unified API
+- **Flexible model configuration** for different providers
+- 
+
+## Quick Start
+
+### Basic Setup
+
+```javascript
+import { router } from 'llmjs2';
+
+// Define your model deployments
+const modelList = [
+  {
+    "model_name": "gpt-3.5-turbo",
+    "llm_params": {
+      "model": "ollama/chatgpt-v-2",
+      "api_key": process.env.OLLAMA_API_KEY,
+      "api_base": process.env.OLLAMA_API_BASE
+    }
+  },
+  {
+    "model_name": "openai-turbo",
+    "llm_params": {
+      "model": "gpt-3.5-turbo",
+      "api_key": process.env.OPENAI_API_KEY
+    }
+  },
+  {
+    "model_name": "gpt-4",
+    "llm_params": {
+      "model": "ollama/gpt-4",
+      "api_key": process.env.OLLAMA_API_KEY,
+      "api_base": process.env.OLLAMA_API_BASE
+    }
+  }
+];
+
+// Create routers with different strategies
+const defaultRouter = router(modelList);
+const randomRouter = router(modelList, 'random');
+const sequentialRouter = router(modelList, 'sequential');
+```
+
+### Basic Usage
+
+```javascript
+// Route to specific model
+const response = await defaultRouter.completion({
+  model: "gpt-3.5-turbo",
+  messages: [{"role": "user", "content": "Hey, how's it going?"}]
+});
+
+// Auto-route with random strategy
+const randomResponse = await randomRouter.completion({
+  messages: [{"role": "user", "content": "Hey, how's it going?"}]
+});
+
+// Auto-route with sequential strategy
+const seqResponse = await sequentialRouter.completion({
+  messages: [{"role": "user", "content": "Hey, how's it going?"}]
+});
+```
+
+## Model Configuration
+
+### Model List Format
+
+Each model in the list is defined with:
+
+```javascript
+{
+  "model_name": "string",     // Alias for routing (can have multiple providers)
+  "llm_params": {             // Provider-specific parameters
+    "model": "string",        // Actual model identifier for the provider
+    "api_key": "string",      // API key (can use environment variables)
+    "api_base": "string?",    // Custom API base URL (optional)
+    // ... other provider-specific params
+  }
+}
+```
+
+### Supported Providers
+
+#### Ollama
+
+```javascript
+{
+  "model_name": "my-ollama-model",
+  "llm_params": {
+    "model": "ollama/llama2",
+    "api_key": process.env.OLLAMA_API_KEY,
+    "api_base": process.env.OLLAMA_API_BASE
+  }
+}
+```
+
+#### OpenRouter
+
+```javascript
+{
+  "model_name": "my-openrouter-model",
+  "llm_params": {
+    "model": "openrouter/free-model",
+    "api_key": process.env.OPEN_ROUTER_API_KEY
+  }
+}
+```
+
+#### OpenAI
+
+```javascript
+{
+  "model_name": "my-openai-model",
+  "llm_params": {
+    "model": "openai/gpt-4",
+    "api_key": process.env.OPENAI_API_KEY
+  }
+}
+```
+
+## Routing Strategies
+
+### Default Strategy
+
+When no strategy is specified, uses load balancing across models with the same `model_name`.
+
+```javascript
+const route = router(modelList); // or router(modelList, 'default')
+
+// Routes to one of the models with model_name="gpt-3.5-turbo"
+const response = await route.completion({
+  model: "gpt-3.5-turbo",
+  messages: [...]
+});
+```
+
+### Random Strategy
+
+Randomly selects from available models when no specific model is requested.
+
+```javascript
+const route = router(modelList, 'random');
+
+// Randomly selects from ALL models in the list
+const response = await route.completion({
+  messages: [...]
+});
+```
+
+### Sequential Strategy
+
+Cycles through models in order for each request.
+
+```javascript
+const route = router(modelList, 'sequential');
+
+// Uses first model, then second, then third, etc.
+const response1 = await route.completion({ messages: [...] }); // model 1
+const response2 = await route.completion({ messages: [...] }); // model 2
+const response3 = await route.completion({ messages: [...] }); // model 3
+// ... cycles back to model 1
+```
+
+## Advanced Usage
+
+### Load Balancing
+
+Multiple models with the same `model_name` enable load balancing:
+
+```javascript
+const modelList = [
+  {
+    "model_name": "gpt-3.5-turbo",
+    "llm_params": {
+      "model": "ollama/chatgpt-v-2",
+      "api_key": process.env.OLLAMA_API_KEY
+    }
+  },
+  {
+    "model_name": "gpt-3.5-turbo",  // Same name - load balancing
+    "llm_params": {
+      "model": "openai/gpt-3.5-turbo",
+      "api_key": process.env.OPENAI_API_KEY
+    }
+  },
+  {
+    "model_name": "gpt-3.5-turbo",  // Same name - load balancing
+    "llm_params": {
+      "model": "openrouter/minimax2.5",
+      "api_key": process.env.OPEN_ROUTER_API_KEY,
+      "api_base": process.env.OPEN_ROUTER_BASE
+    }
+  }
+];
+
+const route = router(modelList);
+
+// This will load balance across all 3 "gpt-3.5-turbo" models
+const response = await route.completion({
+  model: "gpt-3.5-turbo",
+  messages: [...]
+});
+```
+
+### Environment Variables
+
+Use environment variables for configuration:
+
+```javascript
+const modelList = [
+  {
+    "model_name": "production-gpt4",
+    "llm_params": {
+      "model": "openai/gpt-4",
+      "api_key": process.env.OPENAI_API_KEY,
+      "api_base": process.env.OPENAI_API_BASE || "https://api.openai.com/v1"
+    }
+  },
+  {
+    "model_name": "staging-gpt4",
+    "llm_params": {
+      "model": "ollama/gpt-4",
+      "api_key": process.env.OLLAMA_API_KEY,
+      "api_base": process.env.OLLAMA_API_BASE
+    }
+  }
+];
+```
+
+### Complete API Reference
+
+```javascript
+import { router } from 'llmjs2';
+
+// Create router
+const myRouter = router(modelList, strategy);
+
+// Completion with specific model
+const response1 = await myRouter.completion({
+  model: "model_name",           // Optional: specific model to route to
+  messages: [...],               // Required: chat messages
+  tools: [...],                  // Optional: function calling tools
+  // ... other completion params
+});
+
+// Auto-routing completion
+const response2 = await myRouter.completion({
+  messages: [...],               // Required: chat messages
+  // Uses routing strategy when no model specified
+});
+```
+
+## Error Handling
+
+```javascript
+try {
+  const response = await route.completion({
+    model: "non-existent-model",
+    messages: [{"role": "user", "content": "Hello"}]
+  });
+} catch (error) {
+  if (error.message.includes('Model not found')) {
+    console.log('Model not configured in router');
+  } else if (error.message.includes('API key')) {
+    console.log('Provider API key missing');
+  } else {
+    console.log('Routing error:', error.message);
+  }
+}
+```
+
+## Use Cases
+
+### Multi-Provider Fallback
+
+```javascript
+const fallbackModels = [
+  {
+    "model_name": "gpt-4_1",
+    "llm_params": { "model": "openai/gpt-4", "api_key": process.env.OPENAI_API_KEY }
+  },
+  {
+    "model_name": "gpt-4_2",
+    "llm_params": { "model": "ollama/gpt-4", "api_key": process.env.OLLAMA_API_KEY }
+  },
+  {
+    "model_name": "gpt-4_3",
+    "llm_params": { "model": "openrouter/gpt-4", "api_key": process.env.OPEN_ROUTER_API_KEY }
+  }
+];
+
+const route = router(fallbackModels);
+
+// Automatically tries different providers if one fails
+const response = await route.completion({
+  messages: [...]
+});
+```
+
+### Cost Optimization
+
+```javascript
+const costOptimizedModels = [
+  {
+    "model_name": "text-davinci-001",
+    "llm_params": { "model": "ollama/text-davinci-003", "api_key": process.env.OLLAMA_API_KEY }
+  },
+  {
+    "model_name": "text-davinci-002",
+    "llm_params": { "model": "openrouter/text-davinci-003", "api_key": process.env.OPENROUTER_API_KEY }
+  },
+  {
+    "model_name": "text-davinci-003",
+    "llm_params": { "model": "openai/gpt-3.5-turbo", "api_key": process.env.OPENAI_API_KEY }
+  }
+];
+
+const route = router(costOptimizedModels, 'random');
+
+// Load balances across cheaper providers
+const response = await route.completion({
+  model: "text-davinci-003",
+  messages: [...]
+});
+```
+
+### A/B Testing
+
+```javascript
+const abTestModels = [
+  {
+    "model_name": "experiment-a",
+    "llm_params": { "model": "gpt-4", "api_key": process.env.OPENAI_API_KEY }
+  },
+  {
+    "model_name": "experiment-b",
+    "llm_params": { "model": "ollama/gpt-4", "api_key": process.env.OLLAMA_API_KEY }
+  }
+];
+
+const route = router(abTestModels, 'random');
+
+// Randomly routes between experiment variants
+const response = await route.completion({
+  model: "experiment-a",  // or "experiment-b"
+  messages: [...]
+});
+```
+
+## Configuration Examples
+
+### Production Setup
+
+```javascript
+const productionModels = [
+  // Primary OpenAI models
+  { "model_name": "gpt-4", "llm_params": { "model": "gpt-4", "api_key": process.env.OPENAI_API_KEY } },
+  { "model_name": "gpt-3.5-turbo", "llm_params": { "model": "openai/gpt-3.5-turbo", "api_key": process.env.OPENAI_API_KEY } },
+
+  // Fallback Ollama models
+  { "model_name": "gpt-4", "llm_params": { "model": "ollama/gpt-4", "api_key": process.env.OLLAMA_API_KEY } },
+  { "model_name": "gpt-3.5-turbo", "llm_params": { "model": "ollama/gpt-3.5-turbo", "api_key": process.env.OLLAMA_API_KEY } },
+
+  // Cost-effective alternatives
+  { "model_name": "gpt-3.5-turbo", "llm_params": { "model": "openrouter/openrouter/free", "api_key": process.env.OPENROUTER_API_KEY } }
+];
+
+export const productionRouter = router(productionModels);
+export const stagingRouter = router(productionModels, 'sequential');
+```
+
+### Development Setup
+
+```javascript
+const devModels = [
+  // Mock/echo models for testing
+  { "model_name": "echo", "llm_params": { "model": "echo", "api_key": "dev" } },
+
+  // Single provider for consistency
+  { "model_name": "gpt-3.5-turbo", "llm_params": { "model": "ollama/gpt-3.5-turbo", "api_key": process.env.OLLAMA_API_KEY } }
+];
+
+export const devRouter = router(devModels, 'sequential');
+```
+
+This router system provides powerful routing capabilities while maintaining a simple, unified API for LLM completion across multiple providers and deployment strategies.