llm_optimizer 0.1.2 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6a0351ff5590228acf939201d0c7eee71e33ee39a0cd20df33e76187c827ab34
-  data.tar.gz: 6bc7df5aa71407be80ecd07104e1dbff9a25a9ecab7a95cb390261f97fa212e8
+  metadata.gz: c6903d2d4c2163d93ffe8d0d5ad9708d64a8472a430ed9f266c9237e468c8585
+  data.tar.gz: c7270f4717ece6778976f46f1601f9e5d45939e3e7926ea7e3ed05b3b641f413
 SHA512:
-  metadata.gz: e6822ea254300a957c8aa5953d267695ebc2ae4c1e2fa478492d175534bc990eac9bb5bb39127b8418e7306a3f5de9e8124832f9aeac9f723979e92a68babdec
-  data.tar.gz: 4248569dc2a969518142ac2749b6b6e9dc6defedec71e67eeec2480ade3a1aea05261736a2438c994906c05a1d1a1a7991055d48eab9ac0cc33f1344a88a221c
+  metadata.gz: 858cad7443f7adcbe42b3d5ce62b4e815081d2238b7711066276ee2a7c0fb6a506d267ccb48dbe611a2ed08b2eab29139057dcddc2d033155561499a0d6f5421
+  data.tar.gz: b3afc392e8fb2ef5b7baa468f74f9def34a15db9f6df898fd738503638d32f5dda9b04a6c8f2e005cd94aa893eca864111f3be0f2e8bfa1cc0aeef6391e0ae2c

data/CHANGELOG.md

@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.4] - 2026-04-13
+
+### Fixed
+- `WrapperModule#chat` (used by `wrap_client`) incorrectly called `LlmOptimizer.optimize` internally which required `llm_caller` to be configured — causing `ConfigurationError` for users who only called `wrap_client`. Refactored into `optimize_pre_call` / `optimize_post_call` so the wrapped client handles the actual LLM call via `super`. `llm_caller` is no longer needed when using `wrap_client`
+
+### Added
+- `LlmOptimizer.optimize_pre_call(prompt, config)` — runs compress → route → cache lookup without making an LLM call; used internally by `WrapperModule` and available for advanced integrations
+- `LlmOptimizer.optimize_post_call(pre_call_result, response, config)` — stores a response in the semantic cache after an LLM call; used internally by `WrapperModule`
+
+## [0.1.3] - 2026-04-10
+
+### Added
+- `classifier_caller` config option — injectable lambda for LLM-based prompt classification
+- Hybrid routing in `ModelRouter`: fast-path signals (code blocks, keywords) → LLM classifier → word-count heuristic fallback
+- Fixes misclassification of short-but-complex prompts (e.g. "Fix this bug") and long-but-simple prompts
+- Classifier failures (network errors, missing model, unexpected response) automatically fall through to heuristic — no app impact
+- Tests for classifier integration, failure fallback, and fast-path bypass
+
+### Changed
+- `ModelRouter` routing logic now uses three-layer decision chain instead of pure heuristics
+- README updated with classifier documentation and routing decision flow
+
 ## [0.1.2] - 2026-04-10
 
 ### Fixed
@@ -57,7 +79,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `OptimizeResult` struct with `response`, `model`, `model_tier`, `cache_status`, `original_tokens`, `compressed_tokens`, `latency_ms`, `messages`
 - Unit test suite covering all components with positive and negative scenarios using Minitest + Mocha
 
-[Unreleased]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.2...HEAD
+[Unreleased]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.4...HEAD
+[0.1.4]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.3...v0.1.4
+[0.1.3]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.2...v0.1.3
 [0.1.2]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/arunkumarry/llm_optimizer/releases/tag/v0.1.0

data/README.md

@@ -1,6 +1,6 @@
 # llm_optimizer
 
-A Smart Gateway for LLM API calls in Ruby and Rails applications. Reduces token usage and API costs through four composable optimizations — all opt-in, all independently configurable.
+A Smart Gateway for LLM API calls in Ruby and Rails applications. Reduces token usage and API costs through four composable optimizations all opt-in, all independently configurable.
 
 ## How it works
 
@@ -10,17 +10,41 @@ Every call to `LlmOptimizer.optimize` passes through an ordered pipeline:
 prompt → Compressor → ModelRouter → SemanticCache lookup → HistoryManager → LLM call → SemanticCache store → OptimizeResult
 ```
 
-Each stage is independently enabled via configuration flags. If any stage fails, the gem falls through to a raw LLM call — your app never breaks because of the optimizer.
+Each stage is independently enabled via configuration flags. If any stage fails, the gem falls through to a raw LLM call your app never breaks because of the optimizer.
 
 ## Optimizations
 
 ### 1. Semantic Caching
-Stores prompt embeddings in Redis. On subsequent calls, computes cosine similarity against stored embeddings. If similarity ≥ threshold, returns the cached response instantly — no LLM call made.
+Stores prompt embeddings in Redis. On subsequent calls, computes cosine similarity against stored embeddings. If similarity ≥ threshold, returns the cached response instantly no LLM call made.
 
 ### 2. Intelligent Model Routing
-Classifies each prompt using a heuristic and routes it to the appropriate model tier:
-- **Simple** — short prompts (< 20 words), no code blocks, no complex keywords → cheaper/faster model
-- **Complex** — code blocks, keywords like `analyze`, `refactor`, `debug`, `architect`, `explain in detail` → premium model
+
+Classifies each prompt and routes it to the appropriate model tier:
+
+- **Simple** → cheaper/faster model (e.g. `gpt-4o-mini`, `amazon.nova-micro`)
+- **Complex** → premium model (e.g. `claude-3-5-sonnet`, `gpt-4o`)
+
+Routing uses a three-layer decision chain:
+
+1. **Explicit override** — if `route_to: :simple` or `:complex` is set, always use that
+2. **Fast-path signals** — code blocks (` ``` `, `~~~`) and keywords (`analyze`, `refactor`, `debug`, `architect`, `explain in detail`) → instantly `:complex`, no LLM call
+3. **LLM classifier** (optional) — for ambiguous prompts, calls a cheap model with a classification prompt; falls back to word-count heuristic if not configured or if the call fails
+
+This hybrid approach fixes the core weakness of pure heuristics:
+- `"Fix this bug"` → 3 words but `:complex` via classifier
+- `"Explain Ruby blocks simply"` → long but `:simple` via classifier
+- `"analyze this code"` → keyword fast-path → `:complex` instantly (no classifier call)
+
+Configure the classifier with any cheap model your app already uses:
+
+```ruby
+config.classifier_caller = ->(prompt) {
+  RubyLLM.chat(model: "amazon.nova-micro-v1:0", provider: :bedrock, assume_model_exists: true)
+    .ask(prompt).content.strip.downcase
+}
+```
+
+If `classifier_caller` is not set, the router falls back to the word-count heuristic (< 20 words → `:simple`).
 
 ### 3. Token Pruning
 Removes common English stop words from prompts before sending to the LLM. Preserves fenced code block content unchanged. Typically reduces token count by 10–20%.
@@ -120,6 +144,13 @@ LlmOptimizer.configure do |config|
   config.embedding_caller = ->(text) {
     MyEmbeddingService.embed(text)
   }
+
+  # Classifier caller — optional, improves routing accuracy for ambiguous prompts
+  # Falls back to word-count heuristic if not set or if the call fails
+  config.classifier_caller = ->(prompt) {
+    RubyLLM.chat(model: "amazon.nova-micro-v1:0", provider: :bedrock, assume_model_exists: true)
+      .ask(prompt).content.strip.downcase
+  }
 end
 ```
 
@@ -143,6 +174,7 @@ end
 | `debug_logging` | Boolean | `false` | Log full prompt and response at DEBUG level |
 | `llm_caller` | Lambda | `nil` | `(prompt, model:) -> String` |
 | `embedding_caller` | Lambda | `nil` | `(text) -> Array<Float>` |
+| `classifier_caller` | Lambda | `nil` | `(prompt) -> "simple" or "complex"` |
 
 ## Per-call configuration
 
@@ -179,7 +211,7 @@ Transparently wrap an existing LLM client class so all calls through it are auto
 LlmOptimizer.wrap_client(OpenAI::Client)
 ```
 
-This prepends the optimization pipeline into the client's `chat` method. Safe to call multiple times — idempotent.
+This prepends the optimization pipeline into the client's `chat` method. Safe to call multiple times idempotent.
 
 ## OptimizeResult
 

data/lib/generators/llm_optimizer/templates/initializer.rb

@@ -64,5 +64,16 @@ LlmOptimizer.configure do |config|
   # Example:
   #   config.embedding_caller = ->(text) { EmbeddingService.embed(text) }
   #
-  # config.embedding_caller = nil
+  # --- Routing classifier (optional) ---
+  # When set, ambiguous prompts are classified by a cheap LLM instead of
+  # falling back to the word-count heuristic. Unambiguous signals (code blocks,
+  # keywords) still bypass the classifier for speed.
+  #
+  # Example:
+  #   config.classifier_caller = ->(prompt) {
+  #     RubyLLM.chat(model: "amazon.nova-micro-v1:0", assume_model_exists: true)
+  #       .ask(prompt).content.strip.downcase
+  #   }
+  #
+  # config.classifier_caller = nil
 end

data/lib/llm_optimizer/configuration.rb

@@ -21,6 +21,7 @@ module LlmOptimizer
       cache_ttl
       llm_caller
       embedding_caller
+      classifier_caller
     ].freeze
 
     # Define readers for all known keys (setters below track explicit sets)
@@ -45,6 +46,7 @@ module LlmOptimizer
       @cache_ttl            = 86_400
       @llm_caller           = nil
       @embedding_caller     = nil
+      @classifier_caller    = nil
     end
 
     # Copies only explicitly set keys from other_config without resetting unmentioned keys.

data/lib/llm_optimizer/model_router.rb

@@ -6,27 +6,55 @@ module LlmOptimizer
     COMPLEX_PHRASES  = ["explain in detail"].freeze
     CODE_BLOCK_RE    = /```|~~~/
 
+    CLASSIFIER_PROMPT = <<~PROMPT
+      Classify the following prompt as either 'simple' or 'complex'.
+
+      Rules:
+      - simple: factual questions, basic lookups, short explanations, greetings
+      - complex: code generation, debugging, architecture, multi-step reasoning, analysis
+
+      Reply with exactly one word: simple or complex
+
+      Prompt: %<prompt>s
+    PROMPT
+
     def initialize(config)
       @config = config
     end
 
     def route(prompt)
-      # explicit override
+      # Explicit override — always
       return @config.route_to if %i[simple complex].include?(@config.route_to)
 
-      # fenced code block
+      # Unambiguous fast-path signals (no LLM call needed)
       return :complex if CODE_BLOCK_RE.match?(prompt)
 
-      # complex keywords or phrases
       lower = prompt.downcase
       return :complex if COMPLEX_KEYWORDS.any? { |kw| lower.include?(kw) }
-      return :complex if COMPLEX_PHRASES.any? { |ph| lower.include?(ph) }
+      return :complex if COMPLEX_PHRASES.any?  { |ph| lower.include?(ph) }
+
+      # LLM classifier for ambiguous prompts
+      if @config.classifier_caller
+        result = classify_with_llm(prompt)
+        return result if result
+      end
+
+      # Fallback heuristic
+      prompt.split.length < 20 ? :simple : :complex
+    end
+
+    private
 
-      # short prompt
-      return :simple if prompt.split.length < 20
+    def classify_with_llm(prompt)
+      classifier_prompt = format(CLASSIFIER_PROMPT, prompt: prompt)
+      response = @config.classifier_caller.call(classifier_prompt)
+      normalized = response.to_s.strip.downcase.gsub(/[^a-z]/, "")
+      return :simple  if normalized == "simple"
+      return :complex if normalized == "complex"
 
-      # default
-      :complex
+      nil # unrecognized response — fall through to heuristic
+    rescue StandardError
+      nil # classifier failure — fall through to heuristic
     end
   end
 end

data/lib/llm_optimizer/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module LlmOptimizer
-  VERSION = "0.1.2"
+  VERSION = "0.1.4"
 end

data/lib/llm_optimizer.rb

@@ -58,12 +58,37 @@ module LlmOptimizer
   end
 
   # Opt-in client wrapping
+  # WrapperModule intercepts `chat` on the wrapped client, runs the pre-call
+  # optimization pipeline (compress, route, cache lookup), and delegates the
+  # actual LLM call to the original client via `super` — so llm_caller is NOT
+  # required when using wrap_client.
   module WrapperModule
-    def chat(params, &)
+    def chat(params, &block)
+      config = LlmOptimizer.configuration
       prompt = params[:messages] || params[:prompt]
-      optimized = LlmOptimizer.optimize(prompt)
-      params = params.merge(messages: optimized.messages, model: optimized.model)
-      super
+
+      # Run pre-call pipeline: compress, route, cache lookup
+      result = LlmOptimizer.optimize_pre_call(prompt, config)
+
+      # Cache hit — return immediately without calling the LLM
+      if result[:cache_status] == :hit
+        return result[:response]
+      end
+
+      # Apply compressed prompt and routed model, then delegate to original client
+      optimized_params = params.merge(model: result[:model])
+      if params[:messages]
+        optimized_params = optimized_params.merge(messages: result[:prompt])
+      elsif params[:prompt]
+        optimized_params = optimized_params.merge(prompt: result[:prompt])
+      end
+
+      response = super(optimized_params, &block)
+
+      # Store in cache after successful LLM call
+      LlmOptimizer.optimize_post_call(result, response, config)
+
+      response
     end
   end
 
@@ -231,6 +256,53 @@ module LlmOptimizer
     )
   end
 
+  # Pre-call pipeline for wrap_client: compress, route, cache lookup.
+  # Returns a hash with :prompt, :model, :model_tier, :embedding, :cache_status, :response.
+  # Does NOT make an LLM call — the wrapped client handles that via super.
+  def self.optimize_pre_call(prompt, config = configuration)
+    compressor = Compressor.new
+    prompt     = compressor.compress(prompt) if config.compress_prompt
+
+    router     = ModelRouter.new(config)
+    model_tier = router.route(prompt)
+    model      = model_tier == :simple ? config.simple_model : config.complex_model
+
+    embedding = nil
+    if config.use_semantic_cache && config.redis_url
+      begin
+        emb_client = EmbeddingClient.new(
+          model: config.embedding_model,
+          timeout_seconds: config.timeout_seconds,
+          embedding_caller: config.embedding_caller
+        )
+        embedding = emb_client.embed(prompt)
+        redis  = build_redis(config.redis_url)
+        cache  = SemanticCache.new(redis, threshold: config.similarity_threshold, ttl: config.cache_ttl)
+        cached = cache.lookup(embedding)
+        return { prompt: prompt, model: model, model_tier: model_tier,
+                 embedding: embedding, cache_status: :hit, response: cached } if cached
+      rescue EmbeddingError => e
+        config.logger.warn("[llm_optimizer] wrap_client EmbeddingError (cache miss): #{e.message}")
+        embedding = nil
+      end
+    end
+
+    { prompt: prompt, model: model, model_tier: model_tier,
+      embedding: embedding, cache_status: :miss, response: nil }
+  end
+
+  # Post-call: store the LLM response in the semantic cache if applicable.
+  def self.optimize_post_call(pre_call_result, response, config = configuration)
+    return unless config.use_semantic_cache && config.redis_url
+    return unless pre_call_result[:embedding]
+
+    redis = build_redis(config.redis_url)
+    cache = SemanticCache.new(redis, threshold: config.similarity_threshold, ttl: config.cache_ttl)
+    cache.store(pre_call_result[:embedding], response)
+  rescue StandardError => e
+    config.logger.warn("[llm_optimizer] wrap_client cache store failed: #{e.message}")
+  end
+
   # Private helpers
 
   class << self

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm_optimizer
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.4
 platform: ruby
 authors:
 - arun kumar