llm_optimizer 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e68575c7d9fba996b2efeb9dd559635bba72316322b75a1f43b0e6b2a5e28fce
+  data.tar.gz: b8b3f8e06da0d860af65a192b3a4fb16bfe8110fd3eca69d328fd5ef71471571
+SHA512:
+  metadata.gz: 6679cbc09844d71e3c42e74e313d5366bcafbfdeb7e6625a6f0ad591bb8fc98687ba3040a6c54703388bc81be614fd9095d5f71eb0b3eac80fdf0c43299445ee
+  data.tar.gz: '095e4ac8ef5f45240f9d0d5068cf462d66900258765d184f1e25b2e9233c782d5363a248dc178167ad83055fa0d6a006bb2631a83e94c482fcd03fed947c41d4'

data/CHANGELOG.md

@@ -0,0 +1,33 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-04-10
+
+### Added
+
+- `LlmOptimizer.optimize(prompt, options = {}, &block)` — primary entry point returning an `OptimizeResult`
+- `LlmOptimizer.configure` — global configuration with merge semantics (multiple calls merge without resetting)
+- `LlmOptimizer.reset_configuration!` — resets global config to defaults (useful in tests)
+- `LlmOptimizer.wrap_client(client_class)` — opt-in idempotent client wrapping via module prepend
+- **Semantic Caching** — Redis-backed vector similarity cache using cosine similarity; configurable threshold and TTL
+- **Intelligent Model Routing** — heuristic classifier routing prompts to `:simple` or `:complex` model tier based on word count, code blocks, and keywords
+- **Token Pruning / Compressor** — English stop-word removal with fenced code block preservation; `estimate_tokens` helper
+- **Conversation History Sliding Window** — summarizes oldest messages when token budget is exceeded; falls back to original messages on LLM failure
+- **EmbeddingClient** — injectable `embedding_caller` lambda with OpenAI fallback via `OPENAI_API_KEY`
+- **`llm_caller`** — injectable lambda to wire any LLM provider (RubyLLM, ruby-openai, Anthropic, Bedrock, etc.)
+- **Rails generator** — `rails generate llm_optimizer:install` creates a pre-filled initializer
+- **Railtie** — auto-loads generator when used in a Rails app
+- **Structured logging** — INFO log per optimize call (no prompt content); DEBUG log with full prompt/response when `debug_logging: true`
+- **Resilience** — all component failures fall through to raw LLM call; `EmbeddingError` treated as cache miss
+- Full exception hierarchy: `LlmOptimizer::Error`, `ConfigurationError`, `EmbeddingError`, `TimeoutError`
+- `OptimizeResult` struct with `response`, `model`, `model_tier`, `cache_status`, `original_tokens`, `compressed_tokens`, `latency_ms`, `messages`
+- Unit test suite covering all components with positive and negative scenarios using Minitest + Mocha
+
+[Unreleased]: https://github.com/arunkumarry/llm_optimizer/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/arunkumarry/llm_optimizer/releases/tag/v0.1.0

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 arun kumar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,243 @@
+# llm_optimizer
+
+A Smart Gateway for LLM API calls in Ruby and Rails applications. Reduces token usage and API costs through four composable optimizations — all opt-in, all independently configurable.
+
+## How it works
+
+Every call to `LlmOptimizer.optimize` passes through an ordered pipeline:
+
+```
+prompt → Compressor → ModelRouter → SemanticCache lookup → HistoryManager → LLM call → SemanticCache store → OptimizeResult
+```
+
+Each stage is independently enabled via configuration flags. If any stage fails, the gem falls through to a raw LLM call — your app never breaks because of the optimizer.
+
+## Optimizations
+
+### 1. Semantic Caching
+Stores prompt embeddings in Redis. On subsequent calls, computes cosine similarity against stored embeddings. If similarity ≥ threshold, returns the cached response instantly — no LLM call made.
+
+### 2. Intelligent Model Routing
+Classifies each prompt using a heuristic and routes it to the appropriate model tier:
+- **Simple** — short prompts (< 20 words), no code blocks, no complex keywords → cheaper/faster model
+- **Complex** — code blocks, keywords like `analyze`, `refactor`, `debug`, `architect`, `explain in detail` → premium model
+
+### 3. Token Pruning
+Removes common English stop words from prompts before sending to the LLM. Preserves fenced code block content unchanged. Typically reduces token count by 10–20%.
+
+### 4. Conversation History Sliding Window
+When a conversation history exceeds the configured token budget, summarizes the oldest messages using the simple model and replaces them with a single system summary message.
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "llm_optimizer"
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+For Rails apps, generate the initializer:
+
+```bash
+rails generate llm_optimizer:install
+```
+
+This creates `config/initializers/llm_optimizer.rb` with all options pre-filled and commented.
+
+## Quick Start
+
+```ruby
+LlmOptimizer.configure do |config|
+  config.compress_prompt    = true
+  config.use_semantic_cache = true
+  config.redis_url          = ENV["REDIS_URL"]
+
+  # Wire up your app's LLM client
+  config.llm_caller = ->(prompt, model:) {
+    # Use whatever LLM client your app already has
+    MyLlmService.chat(prompt, model: model)
+  }
+
+  # Wire up your embeddings provider (required if use_semantic_cache: true)
+  config.embedding_caller = ->(text) {
+    MyEmbeddingService.embed(text)
+  }
+end
+
+result = LlmOptimizer.optimize("What is Redis?")
+
+puts result.response          # => "Redis is an in-memory data store..."
+puts result.cache_status      # => :hit or :miss
+puts result.model_tier        # => :simple or :complex
+puts result.model             # => "gpt-4o-mini"
+puts result.original_tokens   # => 5
+puts result.compressed_tokens # => 4
+puts result.latency_ms        # => 12.4
+```
+
+## Configuration
+
+### Rails initializer
+
+```ruby
+LlmOptimizer.configure do |config|
+  # Feature flags — all off by default
+  config.compress_prompt    = true   # strip stop words before sending to LLM
+  config.use_semantic_cache = true   # cache responses by vector similarity
+  config.manage_history     = true   # summarize old messages when over token budget
+
+  # Model routing
+  config.route_to      = :auto          # :auto | :simple | :complex
+  config.simple_model  = "gpt-4o-mini"  # model used for simple prompts
+  config.complex_model = "claude-3-5-sonnet-20241022"  # model used for complex prompts
+
+  # Redis (required if use_semantic_cache: true)
+  config.redis_url = ENV["REDIS_URL"]
+
+  # Tuning
+  config.similarity_threshold = 0.96   # cosine similarity cutoff for cache hit (0.0–1.0)
+  config.token_budget         = 4000   # token limit before history summarization
+  config.cache_ttl            = 86400  # cache TTL in seconds (default: 24h)
+  config.timeout_seconds      = 5      # timeout for external API calls
+
+  # Logging
+  config.logger        = Rails.logger
+  config.debug_logging = Rails.env.development?  # logs full prompt+response at DEBUG level
+
+  # LLM caller — wire to your existing LLM client (required)
+  config.llm_caller = ->(prompt, model:) {
+    RubyLLM.chat(model: model, assume_model_exists: true).ask(prompt).content
+  }
+
+  # Embeddings caller — wire to your embeddings provider (required if use_semantic_cache: true)
+  # Falls back to OpenAI via ENV["OPENAI_API_KEY"] if not set
+  config.embedding_caller = ->(text) {
+    MyEmbeddingService.embed(text)
+  }
+end
+```
+
+### Configuration reference
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `compress_prompt` | Boolean | `false` | Strip stop words before sending to LLM |
+| `use_semantic_cache` | Boolean | `false` | Enable Redis-backed semantic cache |
+| `manage_history` | Boolean | `false` | Enable conversation history summarization |
+| `route_to` | Symbol | `:auto` | `:auto`, `:simple`, or `:complex` |
+| `simple_model` | String | `"gpt-4o-mini"` | Model for simple prompts |
+| `complex_model` | String | `"claude-3-5-sonnet-20241022"` | Model for complex prompts |
+| `similarity_threshold` | Float | `0.96` | Minimum cosine similarity for cache hit |
+| `token_budget` | Integer | `4000` | Token limit before history summarization |
+| `cache_ttl` | Integer | `86400` | Cache entry TTL in seconds |
+| `timeout_seconds` | Integer | `5` | Timeout for external API calls |
+| `redis_url` | String | `nil` | Redis connection URL |
+| `embedding_model` | String | `"text-embedding-3-small"` | Embedding model name (OpenAI fallback) |
+| `logger` | Logger | `Logger.new($stdout)` | Any Logger-compatible object |
+| `debug_logging` | Boolean | `false` | Log full prompt and response at DEBUG level |
+| `llm_caller` | Lambda | `nil` | `(prompt, model:) -> String` |
+| `embedding_caller` | Lambda | `nil` | `(text) -> Array<Float>` |
+
+## Per-call configuration
+
+Override global config for a single call using a block:
+
+```ruby
+result = LlmOptimizer.optimize(prompt) do |config|
+  config.route_to      = :simple
+  config.compress_prompt = false
+end
+```
+
+## Conversation history
+
+Pass a `messages` array to enable history management:
+
+```ruby
+messages = [
+  { role: "user",      content: "Tell me about Redis" },
+  { role: "assistant", content: "Redis is an in-memory data store..." },
+  # ... more messages
+]
+
+result = LlmOptimizer.optimize("What else can it do?", messages: messages)
+
+# result.messages contains the (possibly summarized) messages array
+```
+
+## Opt-in client wrapping
+
+Transparently wrap an existing LLM client class so all calls through it are automatically optimized:
+
+```ruby
+LlmOptimizer.wrap_client(OpenAI::Client)
+```
+
+This prepends the optimization pipeline into the client's `chat` method. Safe to call multiple times — idempotent.
+
+## OptimizeResult
+
+Every call returns an `OptimizeResult` struct:
+
+| Field | Type | Description |
+|---|---|---|
+| `response` | String | The LLM response text |
+| `model` | String | Model name actually used |
+| `model_tier` | Symbol | `:simple` or `:complex` |
+| `cache_status` | Symbol | `:hit` or `:miss` |
+| `original_tokens` | Integer | Estimated token count before compression |
+| `compressed_tokens` | Integer | Estimated token count after compression (`nil` if not compressed) |
+| `latency_ms` | Float | Total wall-clock time for the optimize call |
+| `messages` | Array | Final messages array (for history management) |
+
+## Error handling
+
+The gem defines a hierarchy of errors, all inheriting from `LlmOptimizer::Error`:
+
+```
+LlmOptimizer::Error
+├── LlmOptimizer::ConfigurationError  # unknown config key, missing llm_caller
+├── LlmOptimizer::EmbeddingError      # embedding API failure
+└── LlmOptimizer::TimeoutError        # network timeout exceeded
+```
+
+The gateway catches all component failures and falls through to a raw LLM call with the original prompt. Your app's core functionality is never blocked by the optimizer.
+
+## Resilience
+
+| Failure | Behavior |
+|---|---|
+| Redis unavailable (read) | Treat as cache miss, continue |
+| Redis unavailable (write) | Log warning, return LLM result normally |
+| Embedding API failure | Treat as cache miss, continue |
+| Any component exception | Log error, fall through to raw LLM call |
+| History summarization failure | Log error, return original messages unchanged |
+
+## Development
+
+```bash
+bundle install
+bundle exec rake test     # run tests
+bundle exec rake rubocop  # lint
+bundle exec rake          # test + lint
+```
+
+Generate the Rails initializer in a target app:
+
+```bash
+rails generate llm_optimizer:install
+```
+
+## License
+
+MIT
+
+---
+
+[GitHub](https://github.com/arunkumarry/llm_optimizer) · [RubyGems](https://rubygems.org/gems/llm_optimizer) · [Changelog](https://github.com/arunkumarry/llm_optimizer/blob/main/CHANGELOG.md)

data/Rakefile

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+
+Minitest::TestTask.create(:test) do |t|
+  t.libs << "test"
+  t.test_globs = ["test/test_*.rb", "test/unit/test_*.rb"]
+end
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[test rubocop]

data/lib/generators/llm_optimizer/install_generator.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module LlmOptimizer
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Creates a LlmOptimizer initializer in your Rails app"
+
+      def copy_initializer
+        template "initializer.rb", "config/initializers/llm_optimizer.rb"
+      end
+    end
+  end
+end

data/lib/generators/llm_optimizer/templates/initializer.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+# LlmOptimizer initializer
+# Run `rails generate llm_optimizer:install` to regenerate this file.
+#
+# Docs: https://github.com/arunkumar/llm_optimizer
+
+LlmOptimizer.configure do |config|
+  # --- Feature flags ---
+  # All optimizations are off by default. Enable what you need.
+  config.compress_prompt    = false  # strip stop words before sending to LLM
+  config.use_semantic_cache = false  # cache responses by vector similarity in Redis
+  config.manage_history     = false  # summarize old messages when over token budget
+
+  # --- Model routing ---
+  # :auto classifies each prompt; :simple or :complex forces a tier
+  config.route_to      = :auto
+  config.simple_model  = "gpt-4o-mini"
+  config.complex_model = "gpt-4o"
+
+  # --- Redis (required only if use_semantic_cache: true) ---
+  config.redis_url = ENV.fetch("REDIS_URL", nil)
+
+  # --- Tuning ---
+  config.similarity_threshold = 0.96   # cosine similarity cutoff for a cache hit
+  config.token_budget         = 4000   # token limit before history summarization kicks in
+  config.cache_ttl            = 86400  # cache entry TTL in seconds (default: 24h)
+  config.timeout_seconds      = 5      # timeout for embedding / external API calls
+
+  # --- Logging ---
+  config.logger        = Rails.logger
+  config.debug_logging = Rails.env.development?
+
+  # --- LLM caller (required) ---
+  # Wire this up to however your app already calls the LLM.
+  #
+  # Example with ruby-openai:
+  #   config.llm_caller = ->(prompt, model:) {
+  #     OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"])
+  #       .chat(parameters: { model: model, messages: [{ role: "user", content: prompt }] })
+  #       .dig("choices", 0, "message", "content")
+  #   }
+  #
+  # Example with a shared service object:
+  #   config.llm_caller = ->(prompt, model:) {
+  #     provider = if model.include?("claude") then :anthropic
+  #              elsif model.include?("gpt") then :openai
+  #              elsif model.include?("gemini") then :gemini
+  #              elsif model.include?("nova") || model.include?("amazon") then :bedrock
+  #              else :ollama
+  #              end
+  #     RubyLLM.chat(model: model, provider: provider, assume_model_exists: true) }
+  #   end
+  #
+  config.llm_caller = ->(prompt, model:) {
+    raise NotImplementedError, "[llm_optimizer] llm_caller is not configured. " \
+      "Edit config/initializers/llm_optimizer.rb and wire it to your LLM client."
+  }
+
+  # --- Embeddings caller (optional) ---
+  # Only needed if use_semantic_cache: true.
+  # If omitted, falls back to OpenAI via ENV["OPENAI_API_KEY"].
+  #
+  # Example:
+  #   config.embedding_caller = ->(text) { EmbeddingService.embed(text) }
+  #
+  # config.embedding_caller = nil
+end

data/lib/llm_optimizer/compressor.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module LlmOptimizer
+  class Compressor
+    STOP_WORDS = %w[
+      the a an is are was were be been being
+      of in to for on at by with from as into
+      through during before after above below
+      between out off over under again further
+      then once
+    ].freeze
+
+    FENCE_RE = /(```[\s\S]*?```|~~~[\s\S]*?~~~)/
+
+    def initialize(slm_client: nil)
+      @slm_client = slm_client
+    end
+
+    def compress(prompt)
+      segments = prompt.split(FENCE_RE)
+
+      processed = segments.map.with_index do |segment, i|
+        # Odd-indexed segments are fenced code blocks (captured group)
+        if i.odd?
+          segment
+        else
+          remove_stop_words(segment)
+        end
+      end
+
+      result = processed.join
+      result.gsub(/\s{2,}/, " ").strip
+    end
+
+    def estimate_tokens(text)
+      (text.length / 4.0).ceil
+    end
+
+    private
+
+    def remove_stop_words(text)
+      stop_set = STOP_WORDS.to_set
+      words = text.split(" ")
+      words.reject { |w| stop_set.include?(w.downcase) }.join(" ")
+    end
+  end
+end

data/lib/llm_optimizer/configuration.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+require "logger"
+require "set"
+
+module LlmOptimizer
+  class Configuration
+    KNOWN_KEYS = %i[
+      use_semantic_cache
+      compress_prompt
+      manage_history
+      route_to
+      similarity_threshold
+      token_budget
+      redis_url
+      embedding_model
+      simple_model
+      complex_model
+      logger
+      debug_logging
+      timeout_seconds
+      cache_ttl
+      llm_caller
+      embedding_caller
+    ].freeze
+
+    # Define readers for all known keys (setters below track explicit sets)
+    KNOWN_KEYS.each { |key| define_method(key) { instance_variable_get(:"@#{key}") } }
+
+    def initialize
+      @explicitly_set = Set.new
+
+      @use_semantic_cache   = false
+      @compress_prompt      = false
+      @manage_history       = false
+      @route_to             = :auto
+      @similarity_threshold = 0.96
+      @token_budget         = 4000
+      @redis_url            = nil
+      @embedding_model      = "text-embedding-3-small"
+      @simple_model         = "gpt-4o-mini"
+      @complex_model        = "claude-3-5-sonnet-20241022"
+      @logger               = Logger.new($stdout)
+      @debug_logging        = false
+      @timeout_seconds      = 5
+      @cache_ttl            = 86400
+      @llm_caller           = nil
+      @embedding_caller     = nil
+    end
+
+    # Copies only explicitly set keys from other_config without resetting unmentioned keys.
+    def merge!(other_config)
+      other_config.instance_variable_get(:@explicitly_set).each do |key|
+        public_send(:"#{key}=", other_config.public_send(key))
+      end
+      self
+    end
+
+    def method_missing(name, *args, &block)
+      key = name.to_s.chomp("=").to_sym
+      raise ConfigurationError, "Unknown configuration key: #{key}" unless KNOWN_KEYS.include?(key)
+
+      super
+    end
+
+    def respond_to_missing?(name, include_private = false)
+      key = name.to_s.chomp("=").to_sym
+      KNOWN_KEYS.include?(key) || super
+    end
+
+    # Override generated attr_accessor setters to track explicitly set keys.
+    KNOWN_KEYS.each do |key|
+      define_method(:"#{key}=") do |value|
+        @explicitly_set << key
+        instance_variable_set(:"@#{key}", value)
+      end
+    end
+  end
+end

data/lib/llm_optimizer/embedding_client.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require "net/http"
+require "uri"
+require "json"
+
+module LlmOptimizer
+  class EmbeddingClient
+    OPENAI_ENDPOINT = "https://api.openai.com/v1/embeddings"
+
+    def initialize(model:, timeout_seconds:, embedding_caller: nil)
+      @model            = model
+      @timeout_seconds  = timeout_seconds
+      @embedding_caller = embedding_caller
+    end
+
+    def embed(text)
+      if @embedding_caller
+        @embedding_caller.call(text)
+      else
+        embed_via_openai(text)
+      end
+    rescue EmbeddingError
+      raise
+    rescue StandardError => e
+      raise EmbeddingError, "Embedding request failed: #{e.message}"
+    end
+
+    private
+
+    def embed_via_openai(text)
+      api_key = ENV["OPENAI_API_KEY"]
+      raise EmbeddingError, "OPENAI_API_KEY is not set and no embedding_caller configured" if api_key.nil? || api_key.empty?
+
+      uri  = URI(OPENAI_ENDPOINT)
+      body = JSON.generate({ model: @model, input: text })
+
+      http              = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl      = true
+      http.open_timeout = @timeout_seconds
+      http.read_timeout = @timeout_seconds
+
+      request                  = Net::HTTP::Post.new(uri.path)
+      request["Content-Type"]  = "application/json"
+      request["Authorization"] = "Bearer #{api_key}"
+      request.body             = body
+
+      response = http.request(request)
+
+      unless response.is_a?(Net::HTTPSuccess)
+        raise EmbeddingError, "OpenAI embeddings API returned #{response.code}: #{response.body}"
+      end
+
+      parsed = JSON.parse(response.body)
+      parsed.dig("data", 0, "embedding") or
+        raise EmbeddingError, "Unexpected response shape: #{response.body}"
+    rescue Net::OpenTimeout, Net::ReadTimeout => e
+      raise EmbeddingError, "Embedding request timed out: #{e.message}"
+    end
+  end
+end

data/lib/llm_optimizer/history_manager.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module LlmOptimizer
+  class HistoryManager
+    SUMMARIZE_COUNT = 10
+
+    def initialize(llm_caller:, simple_model:, token_budget:)
+      @llm_caller   = llm_caller
+      @simple_model = simple_model
+      @token_budget = token_budget
+    end
+
+    def estimate_tokens(messages)
+      total_chars = messages.sum { |m| (m[:content] || m["content"] || "").length }
+      total_chars / 4
+    end
+
+    def process(messages)
+      return messages if estimate_tokens(messages) <= @token_budget
+
+      count      = [SUMMARIZE_COUNT, messages.length].min
+      to_summarize = messages.first(count)
+      remainder    = messages.drop(count)
+
+      summary = summarize(to_summarize)
+      return messages if summary.nil?
+
+      [{ role: "system", content: summary }] + remainder
+    end
+
+    private
+
+    def summarize(messages)
+      conversation = messages.map { |m| "#{m[:role] || m["role"]}: #{m[:content] || m["content"]}" }.join("\n")
+      prompt = "Summarize the following conversation history concisely, preserving key facts and decisions:\n\n#{conversation}"
+
+      @llm_caller.call(prompt, model: @simple_model)
+    rescue StandardError => e
+      warn "[llm_optimizer] HistoryManager summarization failed: #{e.message}"
+      nil
+    end
+  end
+end

data/lib/llm_optimizer/model_router.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module LlmOptimizer
+  class ModelRouter
+    COMPLEX_KEYWORDS = %w[analyze refactor debug architect].freeze
+    COMPLEX_PHRASES  = ["explain in detail"].freeze
+    CODE_BLOCK_RE    = /```|~~~/
+
+    def initialize(config)
+      @config = config
+    end
+
+    def route(prompt)
+      # explicit override
+      return @config.route_to if @config.route_to == :simple || @config.route_to == :complex
+
+      # fenced code block
+      return :complex if CODE_BLOCK_RE.match?(prompt)
+
+      # complex keywords or phrases
+      lower = prompt.downcase
+      return :complex if COMPLEX_KEYWORDS.any? { |kw| lower.include?(kw) }
+      return :complex if COMPLEX_PHRASES.any? { |ph| lower.include?(ph) }
+
+      # short prompt
+      return :simple if prompt.split.length < 20
+
+      # default
+      :complex
+    end
+  end
+end

data/lib/llm_optimizer/optimize_result.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module LlmOptimizer
+  OptimizeResult = Struct.new(
+    :response, :model, :model_tier, :cache_status,
+    :original_tokens, :compressed_tokens, :latency_ms, :messages,
+    keyword_init: true
+  )
+end

data/lib/llm_optimizer/railtie.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require "rails/railtie"
+
+module LlmOptimizer
+  class Railtie < Rails::Railtie
+    generators do
+      require "generators/llm_optimizer/install_generator"
+    end
+  end
+end

data/lib/llm_optimizer/semantic_cache.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+require "digest"
+require "msgpack"
+
+module LlmOptimizer
+  class SemanticCache
+    KEY_NAMESPACE = "llm_optimizer:cache:"
+
+    def initialize(redis_client, threshold:, ttl:)
+      @redis     = redis_client
+      @threshold = threshold
+      @ttl       = ttl
+    end
+
+    def store(embedding, response)
+      key     = cache_key(embedding)
+      payload = MessagePack.pack({ "embedding" => embedding, "response" => response })
+      @redis.set(key, payload, ex: @ttl)
+    rescue ::Redis::BaseError => e
+      warn "[llm_optimizer] SemanticCache store failed: #{e.message}"
+    end
+
+    def lookup(embedding)
+      keys = @redis.keys("#{KEY_NAMESPACE}*")
+      return nil if keys.empty?
+
+      best_score    = -Float::INFINITY
+      best_response = nil
+
+      keys.each do |key|
+        raw = @redis.get(key)
+        next unless raw
+
+        entry = MessagePack.unpack(raw)
+        stored_embedding = entry["embedding"]
+        score = cosine_similarity(embedding, stored_embedding)
+
+        if score > best_score
+          best_score    = score
+          best_response = entry["response"]
+        end
+      end
+
+      best_score >= @threshold ? best_response : nil
+    rescue ::Redis::BaseError => e
+      warn "[llm_optimizer] SemanticCache lookup failed: #{e.message}"
+      nil
+    end
+
+    def cosine_similarity(vec_a, vec_b)
+      dot    = vec_a.zip(vec_b).sum { |a, b| a * b }
+      mag_a  = Math.sqrt(vec_a.sum { |x| x * x })
+      mag_b  = Math.sqrt(vec_b.sum { |x| x * x })
+      return 0.0 if mag_a.zero? || mag_b.zero?
+
+      dot / (mag_a * mag_b)
+    end
+
+    private
+
+    def cache_key(embedding)
+      KEY_NAMESPACE + Digest::SHA256.hexdigest(embedding.pack("f*"))
+    end
+  end
+end

data/lib/llm_optimizer/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module LlmOptimizer
+  VERSION = "0.1.0"
+end

data/lib/llm_optimizer.rb

@@ -0,0 +1,273 @@
+# frozen_string_literal: true
+
+require_relative "llm_optimizer/version"
+require_relative "llm_optimizer/configuration"
+require_relative "llm_optimizer/optimize_result"
+require_relative "llm_optimizer/compressor"
+require_relative "llm_optimizer/model_router"
+require_relative "llm_optimizer/embedding_client"
+require_relative "llm_optimizer/semantic_cache"
+require_relative "llm_optimizer/history_manager"
+
+require "llm_optimizer/railtie" if defined?(Rails)
+
+module LlmOptimizer
+  # Base error class for all gem-specific exceptions
+  class Error < StandardError; end
+
+  # Raised when an unrecognized configuration key is set
+  class ConfigurationError < Error; end
+
+  # Raised when the embedding API call fails
+  class EmbeddingError < Error; end
+
+  # Raised when a network timeout is exceeded
+  class TimeoutError < Error; end
+
+  # Global configuration
+  @configuration = nil
+
+  # Yields a Configuration instance; merges it into the global config.
+  def self.configure
+    temp = Configuration.new
+    yield temp
+    configuration.merge!(temp)
+    validate_configuration!(configuration)
+  end
+
+  # Warns about misconfigured options rather than failing silently at call time.
+  def self.validate_configuration!(config)
+    if config.use_semantic_cache && config.embedding_caller.nil?
+      config.logger.warn(
+        "[llm_optimizer] use_semantic_cache is true but no embedding_caller is configured. " \
+        "Semantic caching will be skipped. Set config.embedding_caller to enable it."
+      )
+      config.use_semantic_cache = false
+    end
+  end
+
+  # Returns the current global Configuration, lazy-initializing if nil.
+  def self.configuration
+    @configuration ||= Configuration.new
+  end
+
+  # Replaces the global config with a fresh default Configuration.
+  # Useful in tests to avoid state leakage.
+  def self.reset_configuration!
+    @configuration = Configuration.new
+  end
+
+  # Opt-in client wrapping
+  module WrapperModule
+    def chat(params, &block)
+      prompt = params[:messages] || params[:prompt]
+      optimized = LlmOptimizer.optimize(prompt)
+      params = params.merge(messages: optimized.messages, model: optimized.model)
+      super(params, &block)
+    end
+  end
+
+  # Prepends WrapperModule into client_class; idempotent — safe to call N times.
+  def self.wrap_client(client_class)
+    return if client_class.ancestors.include?(WrapperModule)
+
+    client_class.prepend(WrapperModule)
+  end
+
+  # Primary entry point
+  # Runs the optimization pipeline and returns an OptimizeResult.
+
+  # options hash keys mirror Configuration attr_accessors and are merged over
+  # the global config for this call only.  An optional block is yielded a
+  # per-call Configuration for fine-grained control.
+  def self.optimize(prompt, options = {}, &block)
+    start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+    # Resolve per-call configuration — only pass known config keys
+    call_config = Configuration.new
+    call_config.merge!(configuration)
+    options.each do |k, v|
+      next unless LlmOptimizer::Configuration::KNOWN_KEYS.include?(k.to_sym)
+      call_config.public_send(:"#{k}=", v)
+    end
+    yield call_config if block_given?
+
+    logger = call_config.logger
+
+    # Keep a reference to the original prompt for fallback use
+    original_prompt = prompt
+
+    # Compression
+    compressor      = Compressor.new
+    original_tokens = compressor.estimate_tokens(prompt)
+    compressed_tokens = nil
+
+    if call_config.compress_prompt
+      prompt            = compressor.compress(prompt)
+      compressed_tokens = compressor.estimate_tokens(prompt)
+    end
+
+    # Model routing
+    router     = ModelRouter.new(call_config)
+    model_tier = router.route(prompt)
+    model      = model_tier == :simple ? call_config.simple_model : call_config.complex_model
+
+    # Semantic cache lookup
+    embedding = nil
+
+    if call_config.use_semantic_cache
+      begin
+        emb_client = EmbeddingClient.new(
+          model:            call_config.embedding_model,
+          timeout_seconds:  call_config.timeout_seconds,
+          embedding_caller: call_config.embedding_caller
+        )
+        embedding = emb_client.embed(prompt)
+
+        if call_config.redis_url
+          redis  = build_redis(call_config.redis_url)
+          cache  = SemanticCache.new(redis, threshold: call_config.similarity_threshold, ttl: call_config.cache_ttl)
+          cached = cache.lookup(embedding)
+
+          if cached
+            latency_ms = elapsed_ms(start)
+            emit_log(logger, call_config,
+                     cache_status: :hit, model_tier: model_tier,
+                     original_tokens: original_tokens, compressed_tokens: compressed_tokens,
+                     latency_ms: latency_ms, prompt: original_prompt, response: cached)
+            return OptimizeResult.new(
+              response:          cached,
+              model:             model,
+              model_tier:        model_tier,
+              cache_status:      :hit,
+              original_tokens:   original_tokens,
+              compressed_tokens: compressed_tokens,
+              latency_ms:        latency_ms,
+              messages:          options[:messages]
+            )
+          end
+        end
+      rescue EmbeddingError => e
+        logger.warn("[llm_optimizer] EmbeddingError (treating as cache miss): #{e.message}")
+        embedding = nil
+        # continue pipeline as cache miss
+      end
+    end
+
+    # History management
+    messages = options[:messages]
+    if call_config.manage_history && messages
+      llm_caller = ->(p, model:) { raw_llm_call(p, model: model) }
+      history_mgr = HistoryManager.new(
+        llm_caller:   llm_caller,
+        simple_model: call_config.simple_model,
+        token_budget: call_config.token_budget
+      )
+      messages = history_mgr.process(messages)
+    end
+
+    # Raw LLM call
+    response = raw_llm_call(prompt, model: model, config: call_config)
+
+    # Cache store
+    if call_config.use_semantic_cache && embedding && call_config.redis_url
+      begin
+        redis = build_redis(call_config.redis_url)
+        cache = SemanticCache.new(redis, threshold: call_config.similarity_threshold, ttl: call_config.cache_ttl)
+        cache.store(embedding, response)
+      rescue StandardError => e
+        logger.warn("[llm_optimizer] SemanticCache store failed: #{e.message}")
+      end
+    end
+
+    # Build result
+    latency_ms = elapsed_ms(start)
+    emit_log(logger, call_config,
+             cache_status: :miss, model_tier: model_tier,
+             original_tokens: original_tokens, compressed_tokens: compressed_tokens,
+             latency_ms: latency_ms, prompt: original_prompt, response: response)
+
+    OptimizeResult.new(
+      response:          response,
+      model:             model,
+      model_tier:        model_tier,
+      cache_status:      :miss,
+      original_tokens:   original_tokens,
+      compressed_tokens: compressed_tokens,
+      latency_ms:        latency_ms,
+      messages:          messages
+    )
+
+  rescue EmbeddingError => e
+    # Treat embedding failures as cache miss — continue to raw LLM call
+    logger = configuration.logger
+    logger.warn("[llm_optimizer] EmbeddingError (outer rescue, treating as cache miss): #{e.message}")
+    latency_ms = elapsed_ms(start)
+    response   = raw_llm_call(original_prompt, model: nil, config: configuration)
+    OptimizeResult.new(
+      response:          response,
+      model:             nil,
+      model_tier:        nil,
+      cache_status:      :miss,
+      original_tokens:   original_tokens || 0,
+      compressed_tokens: nil,
+      latency_ms:        latency_ms,
+      messages:          options[:messages]
+    )
+
+  rescue LlmOptimizer::Error, StandardError => e
+    logger = configuration.logger
+    logger.error("[llm_optimizer] #{e.class}: #{e.message}\n#{e.backtrace&.first(5)&.join("\n")}")
+    latency_ms = elapsed_ms(start)
+    response   = raw_llm_call(original_prompt, model: nil, config: configuration)
+    OptimizeResult.new(
+      response:          response,
+      model:             nil,
+      model_tier:        nil,
+      cache_status:      :miss,
+      original_tokens:   original_tokens || 0,
+      compressed_tokens: nil,
+      latency_ms:        latency_ms,
+      messages:          options[:messages]
+    )
+  end
+
+  # Private helpers
+
+  class << self
+    private
+
+    def raw_llm_call(prompt, model:, config: nil)
+      caller = config&.llm_caller || @_current_llm_caller
+      raise ConfigurationError,
+            "No llm_caller configured. Set it via LlmOptimizer.configure { |c| c.llm_caller = ->(prompt, model:) { ... } }" unless caller
+
+      caller.call(prompt, model: model)
+    end
+
+    def elapsed_ms(start)
+      ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000).round(2)
+    end
+
+    def emit_log(logger, config, cache_status:, model_tier:, original_tokens:,
+                 compressed_tokens:, latency_ms:, prompt:, response:)
+
+      logger.info(
+        "[llm_optimizer] { cache_status: #{cache_status.inspect}, " \
+        "model_tier: #{model_tier.inspect}, " \
+        "original_tokens: #{original_tokens.inspect}, " \
+        "compressed_tokens: #{compressed_tokens.inspect}, " \
+        "latency_ms: #{latency_ms.inspect} }"
+      )
+
+      if config.debug_logging
+        logger.debug("[llm_optimizer] prompt=#{prompt.inspect} response=#{response.inspect}")
+      end
+    end
+
+    def build_redis(redis_url)
+      require "redis"
+      Redis.new(url: redis_url)
+    end
+  end
+end

data/sig/llm_optimizer.rbs

@@ -0,0 +1,4 @@
+module LlmOptimizer
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,135 @@
+--- !ruby/object:Gem::Specification
+name: llm_optimizer
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- arun kumar
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: msgpack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+- !ruby/object:Gem::Dependency
+  name: logger
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+- !ruby/object:Gem::Dependency
+  name: prop_check
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: mocha
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+description: llm_optimizer reduces LLM API costs by up to 80% through semantic caching
+  (Redis + vector similarity), intelligent model routing, token pruning, and conversation
+  history summarization. Strictly opt-in and non-invasive.
+email:
+- arunr.rubydev@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/generators/llm_optimizer/install_generator.rb
+- lib/generators/llm_optimizer/templates/initializer.rb
+- lib/llm_optimizer.rb
+- lib/llm_optimizer/compressor.rb
+- lib/llm_optimizer/configuration.rb
+- lib/llm_optimizer/embedding_client.rb
+- lib/llm_optimizer/history_manager.rb
+- lib/llm_optimizer/model_router.rb
+- lib/llm_optimizer/optimize_result.rb
+- lib/llm_optimizer/railtie.rb
+- lib/llm_optimizer/semantic_cache.rb
+- lib/llm_optimizer/version.rb
+- sig/llm_optimizer.rbs
+homepage: https://github.com/arunkumarry/llm_optimizer
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://github.com/arunkumarry/llm_optimizer
+  source_code_uri: https://github.com/arunkumarry/llm_optimizer/tree/main
+  changelog_uri: https://github.com/arunkumarry/llm_optimizer/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: Smart Gateway for LLM calls — semantic caching, model routing, token pruning,
+  and history management.
+test_files: []