llm_optimizer 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6a0351ff5590228acf939201d0c7eee71e33ee39a0cd20df33e76187c827ab34
-  data.tar.gz: 6bc7df5aa71407be80ecd07104e1dbff9a25a9ecab7a95cb390261f97fa212e8
+  metadata.gz: ef2fcae7f3d39043f476a555b980685670c65e266f8fc3f9ca4309081d51c066
+  data.tar.gz: c5fb255ad280afba780ea3c417b377ae406dc828178609bf7e21c0bb4f1ba048
 SHA512:
-  metadata.gz: e6822ea254300a957c8aa5953d267695ebc2ae4c1e2fa478492d175534bc990eac9bb5bb39127b8418e7306a3f5de9e8124832f9aeac9f723979e92a68babdec
-  data.tar.gz: 4248569dc2a969518142ac2749b6b6e9dc6defedec71e67eeec2480ade3a1aea05261736a2438c994906c05a1d1a1a7991055d48eab9ac0cc33f1344a88a221c
+  metadata.gz: f84eba0ae06cd7541616c44c8630618eb09f3f8b1d1fe5b588eae285be6dd6a2fcc88f0868a00cbfb91e00b491f56232c0c592b3bbbea579748232a89e8aff1e
+  data.tar.gz: 80fd56954cfa497f2d7c16be68b4c41c6cd01128f3df1e2b1054c3d1005cb869b70317e60ae847dbae2d2f270119812d00d175a4dfa564c447791c2195bc7672

data/CHANGELOG.md

@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.3] - 2026-04-10
+
+### Added
+- `classifier_caller` config option — injectable lambda for LLM-based prompt classification
+- Hybrid routing in `ModelRouter`: fast-path signals (code blocks, keywords) → LLM classifier → word-count heuristic fallback
+- Fixes misclassification of short-but-complex prompts (e.g. "Fix this bug") and long-but-simple prompts
+- Classifier failures (network errors, missing model, unexpected response) automatically fall through to heuristic — no app impact
+- Tests for classifier integration, failure fallback, and fast-path bypass
+
+### Changed
+- `ModelRouter` routing logic now uses three-layer decision chain instead of pure heuristics
+- README updated with classifier documentation and routing decision flow
+
 ## [0.1.2] - 2026-04-10
 
 ### Fixed

data/README.md

@@ -1,6 +1,6 @@
 # llm_optimizer
 
-A Smart Gateway for LLM API calls in Ruby and Rails applications. Reduces token usage and API costs through four composable optimizations — all opt-in, all independently configurable.
+A Smart Gateway for LLM API calls in Ruby and Rails applications. Reduces token usage and API costs through four composable optimizations all opt-in, all independently configurable.
 
 ## How it works
 
@@ -10,17 +10,41 @@ Every call to `LlmOptimizer.optimize` passes through an ordered pipeline:
 prompt → Compressor → ModelRouter → SemanticCache lookup → HistoryManager → LLM call → SemanticCache store → OptimizeResult
 ```
 
-Each stage is independently enabled via configuration flags. If any stage fails, the gem falls through to a raw LLM call — your app never breaks because of the optimizer.
+Each stage is independently enabled via configuration flags. If any stage fails, the gem falls through to a raw LLM call your app never breaks because of the optimizer.
 
 ## Optimizations
 
 ### 1. Semantic Caching
-Stores prompt embeddings in Redis. On subsequent calls, computes cosine similarity against stored embeddings. If similarity ≥ threshold, returns the cached response instantly — no LLM call made.
+Stores prompt embeddings in Redis. On subsequent calls, computes cosine similarity against stored embeddings. If similarity ≥ threshold, returns the cached response instantly no LLM call made.
 
 ### 2. Intelligent Model Routing
-Classifies each prompt using a heuristic and routes it to the appropriate model tier:
-- **Simple** — short prompts (< 20 words), no code blocks, no complex keywords → cheaper/faster model
-- **Complex** — code blocks, keywords like `analyze`, `refactor`, `debug`, `architect`, `explain in detail` → premium model
+
+Classifies each prompt and routes it to the appropriate model tier:
+
+- **Simple** → cheaper/faster model (e.g. `gpt-4o-mini`, `amazon.nova-micro`)
+- **Complex** → premium model (e.g. `claude-3-5-sonnet`, `gpt-4o`)
+
+Routing uses a three-layer decision chain:
+
+1. **Explicit override** — if `route_to: :simple` or `:complex` is set, always use that
+2. **Fast-path signals** — code blocks (` ``` `, `~~~`) and keywords (`analyze`, `refactor`, `debug`, `architect`, `explain in detail`) → instantly `:complex`, no LLM call
+3. **LLM classifier** (optional) — for ambiguous prompts, calls a cheap model with a classification prompt; falls back to word-count heuristic if not configured or if the call fails
+
+This hybrid approach fixes the core weakness of pure heuristics:
+- `"Fix this bug"` → 3 words but `:complex` via classifier ✓
+- `"Explain Ruby blocks simply"` → long but `:simple` via classifier ✓
+- `"analyze this code"` → keyword fast-path → `:complex` instantly (no classifier call) ✓
+
+Configure the classifier with any cheap model your app already uses:
+
+```ruby
+config.classifier_caller = ->(prompt) {
+  RubyLLM.chat(model: "amazon.nova-micro-v1:0", provider: :bedrock, assume_model_exists: true)
+    .ask(prompt).content.strip.downcase
+}
+```
+
+If `classifier_caller` is not set, the router falls back to the word-count heuristic (< 20 words → `:simple`).
 
 ### 3. Token Pruning
 Removes common English stop words from prompts before sending to the LLM. Preserves fenced code block content unchanged. Typically reduces token count by 10–20%.
@@ -120,6 +144,13 @@ LlmOptimizer.configure do |config|
   config.embedding_caller = ->(text) {
     MyEmbeddingService.embed(text)
   }
+
+  # Classifier caller — optional, improves routing accuracy for ambiguous prompts
+  # Falls back to word-count heuristic if not set or if the call fails
+  config.classifier_caller = ->(prompt) {
+    RubyLLM.chat(model: "amazon.nova-micro-v1:0", provider: :bedrock, assume_model_exists: true)
+      .ask(prompt).content.strip.downcase
+  }
 end
 ```
 
@@ -143,6 +174,7 @@ end
 | `debug_logging` | Boolean | `false` | Log full prompt and response at DEBUG level |
 | `llm_caller` | Lambda | `nil` | `(prompt, model:) -> String` |
 | `embedding_caller` | Lambda | `nil` | `(text) -> Array<Float>` |
+| `classifier_caller` | Lambda | `nil` | `(prompt) -> "simple" or "complex"` |
 
 ## Per-call configuration
 
@@ -179,7 +211,7 @@ Transparently wrap an existing LLM client class so all calls through it are auto
 LlmOptimizer.wrap_client(OpenAI::Client)
 ```
 
-This prepends the optimization pipeline into the client's `chat` method. Safe to call multiple times — idempotent.
+This prepends the optimization pipeline into the client's `chat` method. Safe to call multiple times idempotent.
 
 ## OptimizeResult
 

data/lib/generators/llm_optimizer/templates/initializer.rb

@@ -64,5 +64,16 @@ LlmOptimizer.configure do |config|
   # Example:
   #   config.embedding_caller = ->(text) { EmbeddingService.embed(text) }
   #
-  # config.embedding_caller = nil
+  # --- Routing classifier (optional) ---
+  # When set, ambiguous prompts are classified by a cheap LLM instead of
+  # falling back to the word-count heuristic. Unambiguous signals (code blocks,
+  # keywords) still bypass the classifier for speed.
+  #
+  # Example:
+  #   config.classifier_caller = ->(prompt) {
+  #     RubyLLM.chat(model: "amazon.nova-micro-v1:0", assume_model_exists: true)
+  #       .ask(prompt).content.strip.downcase
+  #   }
+  #
+  # config.classifier_caller = nil
 end

data/lib/llm_optimizer/configuration.rb

@@ -21,6 +21,7 @@ module LlmOptimizer
       cache_ttl
       llm_caller
       embedding_caller
+      classifier_caller
     ].freeze
 
     # Define readers for all known keys (setters below track explicit sets)
@@ -45,6 +46,7 @@ module LlmOptimizer
       @cache_ttl            = 86_400
       @llm_caller           = nil
       @embedding_caller     = nil
+      @classifier_caller    = nil
     end
 
     # Copies only explicitly set keys from other_config without resetting unmentioned keys.

data/lib/llm_optimizer/model_router.rb

@@ -6,27 +6,55 @@ module LlmOptimizer
     COMPLEX_PHRASES  = ["explain in detail"].freeze
     CODE_BLOCK_RE    = /```|~~~/
 
+    CLASSIFIER_PROMPT = <<~PROMPT
+      Classify the following prompt as either 'simple' or 'complex'.
+
+      Rules:
+      - simple: factual questions, basic lookups, short explanations, greetings
+      - complex: code generation, debugging, architecture, multi-step reasoning, analysis
+
+      Reply with exactly one word: simple or complex
+
+      Prompt: %<prompt>s
+    PROMPT
+
     def initialize(config)
       @config = config
     end
 
     def route(prompt)
-      # explicit override
+      # Explicit override — always
       return @config.route_to if %i[simple complex].include?(@config.route_to)
 
-      # fenced code block
+      # Unambiguous fast-path signals (no LLM call needed)
       return :complex if CODE_BLOCK_RE.match?(prompt)
 
-      # complex keywords or phrases
       lower = prompt.downcase
       return :complex if COMPLEX_KEYWORDS.any? { |kw| lower.include?(kw) }
-      return :complex if COMPLEX_PHRASES.any? { |ph| lower.include?(ph) }
+      return :complex if COMPLEX_PHRASES.any?  { |ph| lower.include?(ph) }
+
+      # LLM classifier for ambiguous prompts
+      if @config.classifier_caller
+        result = classify_with_llm(prompt)
+        return result if result
+      end
+
+      # Fallback heuristic
+      prompt.split.length < 20 ? :simple : :complex
+    end
+
+    private
 
-      # short prompt
-      return :simple if prompt.split.length < 20
+    def classify_with_llm(prompt)
+      classifier_prompt = format(CLASSIFIER_PROMPT, prompt: prompt)
+      response = @config.classifier_caller.call(classifier_prompt)
+      normalized = response.to_s.strip.downcase.gsub(/[^a-z]/, "")
+      return :simple  if normalized == "simple"
+      return :complex if normalized == "complex"
 
-      # default
-      :complex
+      nil # unrecognized response — fall through to heuristic
+    rescue StandardError
+      nil # classifier failure — fall through to heuristic
     end
   end
 end

data/lib/llm_optimizer/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module LlmOptimizer
-  VERSION = "0.1.2"
+  VERSION = "0.1.3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm_optimizer
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - arun kumar