rubyllm-semantic_router 0.1.0 → 0.4.0

data/README.md

@@ -1,261 +1,218 @@
 # RubyLLM Semantic Router
 
-Route user messages to specialized LLM agents based on semantic similarity. Think of it as a fast, embedding-based classifier that decides which expert should handle each message.
+Route user messages to specialized LLM agents based on semantic similarity.
 
-## The Problem
+## Installation
 
-You have multiple specialized chat agents:
-- A **product expert** that knows your catalog
-- An **account manager** that handles billing and settings
-- A **support agent** that troubleshoots issues
-
-How do you decide which one handles "I can't log in" vs "What's your return policy" vs "Show me laptops under $1000"?
-
-This gem provides fast, embedding-based routing - no LLM call needed for the routing decision itself.
-
-## How It Works
-
-```
-User: "What's your cheapest laptop?"
-            │
-            ▼
-   ┌─────────────────┐
-   │  Embed message  │  ← ~2ms, $0.00001
-   └────────┬────────┘
-            │
-            ▼
-   ┌─────────────────┐
-   │  Find similar   │  ← Compare to your examples
-   │  examples (kNN) │     "Show me computers" → product
-   └────────┬────────┘     "Reset password" → account
-            │
-            ▼
-   ┌─────────────────┐
-   │  Route to       │  ← Product agent handles it
-   │  Product Agent  │
-   └─────────────────┘
+```ruby
+gem 'rubyllm-semantic_router'
 ```
 
-**Key insight**: The routing decision is just an embedding + kNN lookup. No LLM call needed. Fast and cheap.
-
 ## Quick Start
 
 ```ruby
 require 'rubyllm/semantic_router'
 
-# 1. Define your agents as regular RubyLLM chat objects
-product_chat = RubyLLM.chat(model: "gpt-4o-mini")
-                      .with_instructions("You're a product expert. Help users find products.")
+# Create agents as RubyLLM chat objects
+product = RubyLLM.chat(model: "gpt-4o-mini")
+                 .with_instructions("You're a product expert.")
 
-support_chat = RubyLLM.chat(model: "gpt-4o")
-                      .with_instructions("You're technical support. Troubleshoot issues.")
-                      .with_tools(DiagnosticTool, TicketCreator)
+support = RubyLLM.chat(model: "gpt-4o")
+                 .with_instructions("You're technical support.")
 
-# 2. Create router with your agents
+# Create router
 router = RubyLLM::SemanticRouter.new(
-  agents: {
-    product: product_chat,
-    support: support_chat
-  },
-  default_agent: :product  # Fallback when uncertain
+  agents: { product: product, support: support },
+  default_agent: :product
 )
 
-# 3. Train with examples (the more, the better)
+# Add training examples
 router.import_examples([
   { text: "Show me laptops", agent: :product },
-  { text: "Compare these two phones", agent: :product },
-  { text: "What's on sale?", agent: :product },
   { text: "I can't log in", agent: :support },
-  { text: "App keeps crashing", agent: :support },
-  { text: "Error message when I checkout", agent: :support },
 ])
 
-# 4. Chat! Routing happens automatically.
-router.ask("What gaming laptops do you have?")  # → product agent
-router.ask("My order is stuck")                  # → support agent
+# Chat - routing happens automatically
+router.ask("What gaming laptops do you have?")  # → product
+router.ask("My order is stuck")                  # → support
 ```
 
-## When To Use This
+## How It Works
 
-**Good fit:**
-- High-volume customer service with 3+ clearly separated domains
-- Different models per task (cheap for FAQ, expensive for reasoning)
-- Compliance requirements that need audit trails per agent
-- Tool sets that would confuse a single LLM if combined
+1. User sends a message
+2. Router embeds the message (~2ms, ~$0.00001)
+3. Finds similar examples using kNN
+4. Routes to the matching agent
+5. Agent responds with full conversation history
 
-**Probably overkill:**
-- Small apps with <1000 daily users
-- Overlapping domains where context matters more than classification
-- No training examples available
+No LLM call needed for routing - just embeddings.
 
-## API
+## Options
 
-### Defining Agents
+```ruby
+router = RubyLLM::SemanticRouter.new(
+  agents: { ... },
+  default_agent: :product,
+  similarity_threshold: 0.3,          # Route only if confidence > threshold
+  fallback: :default_agent,           # :default_agent | :keep_current | :ask_clarification
+  embedding_model: "text-embedding-3-small",
+  max_words: 50,                      # Truncate messages to first N words (default: unlimited)
+  logger: Rails.logger,               # Enable debug logging (default: nil)
+  cache_ttl: 300,                     # Cache embeddings for 5 minutes (default: nil)
+  max_retries: 3,                     # Retry failed embedding calls (default: 3)
+  retry_base_delay: 0.5               # Base delay for exponential backoff (default: 0.5s)
+)
+```
 
-Agents are just RubyLLM chat objects - use the same API you already know:
+## Debugging
 
 ```ruby
-my_agent = RubyLLM.chat(model: "claude-sonnet-4")
-                  .with_instructions("You're a specialist...")
-                  .with_tools(Tool1, Tool2)
-                  .with_temperature(0.7)
+# Preview without sending
+decision = router.match("test message")
+decision.agent       # => :product
+decision.confidence  # => 0.85
+
+# Detailed routing info
+router.debug_routing("test message")
 ```
 
-### Router Options
+## Batch Routing
+
+Route multiple messages efficiently with a single embedding API call:
 
 ```ruby
-router = RubyLLM::SemanticRouter.new(
-  agents: {
-    product: product_chat,
-    support: support_chat
-  },
-  default_agent: :product,
+messages = [
+  "Show me products",
+  "I need help with my account",
+  "What's your return policy?"
+]
 
-  # When confidence is below threshold, what to do?
-  fallback: :default_agent,      # Use default (default)
-  # fallback: :keep_current,     # Stay with current agent
-  # fallback: :ask_clarification # Ask user to rephrase
+decisions = router.ask_batch(messages)
+# => [RoutingDecision, RoutingDecision, RoutingDecision]
 
-  similarity_threshold: 0.7,     # 0.0-1.0, higher = stricter
-  embedding_model: "text-embedding-3-small"
-)
+decisions.each do |decision|
+  puts "#{decision.agent}: confidence #{decision.confidence}"
+end
 ```
 
-### Training
+## Error Handling
+
+### Configuration Validation
+
+All configuration values are validated. Invalid values raise `ConfigurationError`:
 
 ```ruby
-# One at a time
-router.add_example("Cancel my subscription", agent: :billing)
+# These will raise ConfigurationError:
+router = RubyLLM::SemanticRouter.new(
+  agents: agents,
+  default_agent: :product,
+  similarity_threshold: 1.5  # Must be 0.0-1.0
+)
 
-# Batch import (faster - single embedding API call)
-router.import_examples([
-  { text: "...", agent: :billing },
-  { text: "...", agent: :support },
-])
+RubyLLM::SemanticRouter.configure do |config|
+  config.default_k_neighbors = 0  # Must be positive integer
+end
 ```
 
-### Debugging
+**Validation rules:**
+- `similarity_threshold`: Must be between 0.0 and 1.0
+- `k_neighbors`: Must be a positive integer
+- `max_words`: Must be `nil` or a positive integer
+- `fallback`: Must be `:default_agent`, `:keep_current`, or `:ask_clarification`
+- `cache_ttl`: Must be `nil` or a positive number
+- `max_retries`: Must be a non-negative integer
 
-```ruby
-# Preview routing without sending message
-decision = router.match("test message")
-decision.agent       # => :product
-decision.confidence  # => 0.85
+### Embedding Errors
 
-# See all matches and scores
-router.debug_routing("test message")
-# => {
-#   message: "test message",
-#   threshold: 0.7,
-#   would_route_to: :product,
-#   top_matches: [
-#     { agent: :product, example: "show products", confidence: 0.85 },
-#     { agent: :support, example: "help me", confidence: 0.42 }
-#   ]
-# }
+Failed embedding API calls raise `EmbeddingError` after exhausting retries:
+
+```ruby
+begin
+  router.ask("Hello")
+rescue RubyLLM::SemanticRouter::EmbeddingError => e
+  puts "Embedding failed: #{e.message}"
+end
 ```
 
-### Conversation Flow
+## Global Configuration
+
+Set defaults for all routers:
 
 ```ruby
-router.ask("Show me phones")        # Routes to :product
-router.current_agent                 # => :product
+RubyLLM::SemanticRouter.configure do |config|
+  config.default_embedding_model = "text-embedding-3-small"
+  config.default_similarity_threshold = 0.3
+  config.default_k_neighbors = 3
+  config.default_fallback = :default_agent
+  config.default_max_words = nil
+  config.logger = Rails.logger
+  config.cache_ttl = 300              # 5 minute cache
+  config.max_retries = 3
+  config.retry_base_delay = 0.5
+end
+```
+
+## Storage Options
 
-router.ask("Actually, I need help") # Routes to :support
-router.current_agent                 # => :support
+### In-Memory (default)
 
-# Full history is preserved across agent switches
-router.messages.size                 # => 4 (2 exchanges)
+```ruby
+router.add_example("Show products", agent: :product)
+router.import_examples([...])
 ```
 
-### ActiveRecord + pgvector
+### ActiveRecord + neighbor gem
 
-Use the [neighbor](https://github.com/ankane/neighbor) gem for PostgreSQL:
+Works with PostgreSQL (pgvector), SQLite (sqlite-vec), MySQL (vector), and [more](https://github.com/ankane/neighbor):
 
 ```ruby
-# Migration
-create_table :routing_examples do |t|
-  t.string :agent_name, null: false
-  t.text :example_text, null: false
-  t.vector :embedding, limit: 1536  # text-embedding-3-small dimensions
-end
-
-# Model
 class RoutingExample < ApplicationRecord
   has_neighbors :embedding
 end
 
-# Usage
-router = RubyLLM::SemanticRouter.new(
-  agents: { product: product_chat, support: support_chat },
-  default_agent: :product
-)
 router.with_examples(RoutingExample.all)
-
-# Scoped for multi-tenant
 router.with_examples(RoutingExample.where(tenant_id: current_tenant.id))
 ```
 
-### Custom Vector Search
+### Multi-tenant Scoping
 
-Bring your own vector database (Pinecone, Qdrant, OpenSearch, etc.):
+For multi-tenant applications, use the `scope` parameter to isolate routing examples:
 
 ```ruby
+# Create scoped router
 router = RubyLLM::SemanticRouter.new(
-  agents: { product: product_chat, support: support_chat },
+  agents: { product: product, support: support },
   default_agent: :product,
-  find_examples: ->(embedding, limit:) {
-    # Pinecone
-    Pinecone.index("examples").query(vector: embedding, top_k: limit).matches.map do |m|
-      { agent_name: m.metadata[:agent], text: m.metadata[:text], score: m.score }
-    end
-  }
+  scope: "tenant_123"
 )
 
-# Or with OpenSearch/Searchkick
+# With ActiveRecord, add a router_scope column to your model
+class RoutingExample < ApplicationRecord
+  has_neighbors :embedding
+end
+
+# Examples are automatically filtered by scope
+router.with_examples(RoutingExample.all)  # Only queries where router_scope = "tenant_123"
+```
+
+For in-memory examples, the router filters examples that respond to `router_scope` and match the configured scope.
+
+### Custom Vector Database
+
+```ruby
 router = RubyLLM::SemanticRouter.new(
   agents: { ... },
   default_agent: :product,
   find_examples: ->(embedding, limit:) {
-    RoutingExample.search("*",
-      knn: { field: :embedding, vector: embedding, k: limit }
-    ).map { |r| { agent_name: r.agent_name, text: r.text, distance: r.distance } }
+    # Pinecone, Qdrant, OpenSearch, etc.
+    YourVectorDB.search(embedding, limit: limit).map do |result|
+      { agent_name: result.agent, score: result.score }
+    end
   }
 )
 ```
 
-Return an array of hashes/objects with:
-- `agent_name` (or `agent`) - which agent this example routes to
-- `text` or `example_text` - the example text (optional, for debugging)
-- `distance` (lower is better) or `score` (higher is better)
-
-## How Agents Share Context
-
-When the router switches agents, the new agent sees the **full conversation history** but with its own system prompt. This means:
-
-1. Agent A responds with context
-2. User asks something in Agent B's domain
-3. Router switches to Agent B
-4. Agent B sees the full chat, responds with its own expertise
-
-The conversation flows naturally. Users don't notice the switch.
-
-## Caveats
-
-1. **You need training examples.** At least 5-10 per agent, more is better.
-
-2. **Embeddings aren't magic.** "I want to return this" and "What's your return policy" are different intents. Train for both.
-
-3. **Threshold tuning matters.** Start with 0.7, use `debug_routing` to see scores, adjust.
-
-4. **Tool cycles are atomic.** If Agent A calls a tool, it keeps control until done. No mid-tool handoffs.
-
-## Development
-
-```bash
-bundle install
-bundle exec rspec
-```
+Return hashes with `agent_name`, and either `distance` (lower=better) or `score` (higher=better).
 
 ## License
 

data/lib/rubyllm/semantic_router/configuration.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "logger"
+
 module RubyLLM
   module SemanticRouter
     # Global configuration for the semantic router
@@ -7,20 +9,114 @@ module RubyLLM
       # Default embedding model to use when not specified per-router
       attr_accessor :default_embedding_model
 
+      # Logger instance for debug output (nil = no logging)
+      attr_accessor :logger
+
+      # Embedding cache TTL in seconds (nil = no caching)
+      attr_reader :cache_ttl
+
+      # Maximum retry attempts for embedding API failures
+      attr_reader :max_retries
+
+      # Base delay in seconds for exponential backoff
+      attr_reader :retry_base_delay
+
+      # Default fallback behavior (:default_agent, :keep_current, :ask_clarification)
+      attr_reader :default_fallback
+
       # Default similarity threshold (0.0 - 1.0)
-      attr_accessor :default_similarity_threshold
+      attr_reader :default_similarity_threshold
 
       # Default number of neighbors to consider for routing
-      attr_accessor :default_k_neighbors
+      attr_reader :default_k_neighbors
 
-      # Default fallback behavior (:default_agent, :keep_current, :ask_clarification)
-      attr_accessor :default_fallback
+      # Default max words to use for embedding (nil = unlimited)
+      attr_reader :default_max_words
+
+      VALID_FALLBACKS = %i[default_agent keep_current ask_clarification].freeze
 
       def initialize
         @default_embedding_model = "text-embedding-3-small"
-        @default_similarity_threshold = 0.7
+        @default_similarity_threshold = 0.3
         @default_k_neighbors = 3
         @default_fallback = :default_agent
+        @default_max_words = nil
+        @logger = nil
+        @cache_ttl = nil
+        @max_retries = 3
+        @retry_base_delay = 0.5
+      end
+
+      def cache_ttl=(value)
+        return @cache_ttl = nil if value.nil?
+        unless value.is_a?(Numeric) && value.positive?
+          raise ConfigurationError, "cache_ttl must be nil or a positive number, got: #{value.inspect}"
+        end
+
+        @cache_ttl = value
+      end
+
+      def max_retries=(value)
+        unless value.is_a?(Integer) && value >= 0
+          raise ConfigurationError, "max_retries must be a non-negative integer, got: #{value.inspect}"
+        end
+
+        @max_retries = value
+      end
+
+      def retry_base_delay=(value)
+        unless value.is_a?(Numeric) && value.positive?
+          raise ConfigurationError, "retry_base_delay must be a positive number, got: #{value.inspect}"
+        end
+
+        @retry_base_delay = value
+      end
+
+      def default_similarity_threshold=(value)
+        validate_similarity_threshold!(value)
+        @default_similarity_threshold = value
+      end
+
+      def default_k_neighbors=(value)
+        validate_k_neighbors!(value)
+        @default_k_neighbors = value
+      end
+
+      def default_max_words=(value)
+        validate_max_words!(value)
+        @default_max_words = value
+      end
+
+      def default_fallback=(value)
+        validate_fallback!(value)
+        @default_fallback = value
+      end
+
+      private
+
+      def validate_similarity_threshold!(value)
+        return if value.is_a?(Numeric) && value >= 0.0 && value <= 1.0
+
+        raise ConfigurationError, "similarity_threshold must be a number between 0.0 and 1.0, got: #{value.inspect}"
+      end
+
+      def validate_k_neighbors!(value)
+        return if value.is_a?(Integer) && value.positive?
+
+        raise ConfigurationError, "k_neighbors must be a positive integer, got: #{value.inspect}"
+      end
+
+      def validate_max_words!(value)
+        return if value.nil?
+        return if value.is_a?(Integer) && value.positive?
+
+        raise ConfigurationError, "max_words must be nil or a positive integer, got: #{value.inspect}"
+      end
+
+      def validate_fallback!(value)
+        return if VALID_FALLBACKS.include?(value)
+
+        raise ConfigurationError, "fallback must be one of #{VALID_FALLBACKS.join(', ')}, got: #{value.inspect}"
       end
     end
   end

data/lib/rubyllm/semantic_router/embedding_cache.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module SemanticRouter
+    # Simple in-memory cache for embeddings with TTL support
+    class EmbeddingCache
+      CacheEntry = Struct.new(:embedding, :expires_at, keyword_init: true)
+
+      def initialize(ttl:)
+        @ttl = ttl
+        @cache = {}
+        @mutex = Mutex.new
+      end
+
+      # Get embedding from cache
+      # @param key [String] Cache key (typically the text that was embedded)
+      # @return [Array, nil] Cached embedding or nil if not found/expired
+      def get(key)
+        @mutex.synchronize do
+          entry = @cache[key]
+          return nil unless entry
+          return nil if entry.expires_at < Time.now
+
+          entry.embedding
+        end
+      end
+
+      # Store embedding in cache
+      # @param key [String] Cache key
+      # @param embedding [Array] The embedding vector to cache
+      def set(key, embedding)
+        @mutex.synchronize do
+          @cache[key] = CacheEntry.new(
+            embedding: embedding,
+            expires_at: Time.now + @ttl
+          )
+        end
+      end
+
+      # Get embedding from cache or compute it
+      # @param key [String] Cache key
+      # @yield Block that computes the embedding if not cached
+      # @return [Array] The embedding
+      def fetch(key)
+        cached = get(key)
+        return cached if cached
+
+        embedding = yield
+        set(key, embedding)
+        embedding
+      end
+
+      # Remove expired entries from cache
+      def cleanup!
+        @mutex.synchronize do
+          now = Time.now
+          @cache.delete_if { |_, entry| entry.expires_at < now }
+        end
+      end
+
+      # Clear all cached entries
+      def clear!
+        @mutex.synchronize do
+          @cache.clear
+        end
+      end
+
+      # Number of entries in cache
+      def size
+        @mutex.synchronize { @cache.size }
+      end
+    end
+  end
+end

data/lib/rubyllm/semantic_router/errors.rb

@@ -55,5 +55,12 @@ module RubyLLM
         super("Invalid agent definition: #{message}")
       end
     end
+
+    # Raised when configuration value is invalid
+    class ConfigurationError < Error
+      def initialize(message)
+        super("Invalid configuration: #{message}")
+      end
+    end
   end
 end