semantic-cache 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0a16f649e20d3989d12d16c4e52b809c7f91fc81383d83cdd9e4b86306ceb8d4
+  data.tar.gz: 8e1a0cf3bfdec291aa24141ffdd4c32f31f4d33802e10b4bc08e36b5576daec1
+SHA512:
+  metadata.gz: 339b424cff6654e37aa888b22ef9a9ab8f06d69fcda0d3ed42f62ded02d852d29ac18a3a1ee570539231e8cbdd83922ef909cdc24eb11e301859f73906ca774c
+  data.tar.gz: 857a0e0013fb7d93c006c87418c9adad1fbd0cfb79faa92cadcbf8a35ed9c690e3c26f02df2247d62e6a48254959e29bffc8457a6bef55e3c6807f624d54502c

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [0.1.0] - 2025-01-29
+
+### Added
+
+- Core semantic caching with cosine similarity matching
+- In-memory and Redis cache stores
+- Embedding generation via OpenAI `text-embedding-3-small`
+- Configurable similarity threshold (default: 0.85)
+- TTL-based and tag-based cache invalidation
+- Cost tracking and savings reports
+- Multi-model support (OpenAI, Anthropic, Gemini)
+- Client wrapper / middleware pattern
+- Rails integration (concern + around_action helper)
+- Thread-safe statistics tracking
+- Comprehensive test suite

data/Gemfile

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gemspec
+
+group :development, :test do
+  gem "redis", ">= 4.0"
+end

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 stokry
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,307 @@
+# SemanticCache
+
+**Semantic caching for LLM API calls. Save 70%+ on costs.**
+
+Cache LLM responses using semantic similarity matching. Similar questions return cached answers instantly, cutting API costs dramatically.
+
+```ruby
+cache = SemanticCache.new
+
+# First call — hits the API
+response = cache.fetch("What's the capital of France?") do
+  openai.chat(messages: [{ role: "user", content: "What's the capital of France?" }])
+end
+
+# Second call — semantically similar, returns cached response instantly
+response = cache.fetch("What is France's capital city?") do
+  openai.chat(messages: [{ role: "user", content: "What is France's capital city?" }])
+end
+# => CACHE HIT! No API call.
+```
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "semantic-cache"
+```
+
+Then:
+
+```bash
+bundle install
+```
+
+Or install directly:
+
+```bash
+gem install semantic-cache
+```
+
+## Quick Start
+
+```ruby
+require "semantic_cache"
+
+# Configure (or set OPENAI_API_KEY env var)
+SemanticCache.configure do |c|
+  c.openai_api_key = "sk-..."
+  c.similarity_threshold = 0.85  # How similar queries must be to match (0.0-1.0)
+end
+
+cache = SemanticCache.new
+
+response = cache.fetch("What is Ruby?", model: "gpt-4o") do
+  openai.chat(parameters: {
+    model: "gpt-4o",
+    messages: [{ role: "user", content: "What is Ruby?" }]
+  })
+end
+
+# Check stats
+puts cache.current_stats
+# => { hits: 0, misses: 1, hit_rate: 0.0, savings: "$0.00", ... }
+```
+
+## How It Works
+
+1. Your query is converted to an embedding vector via OpenAI's `text-embedding-3-small`
+2. The cache searches for stored entries with high cosine similarity
+3. If a match exceeds the threshold (default 0.85), the cached response is returned
+4. If no match, the block executes, and the result is cached for future queries
+
+## Configuration
+
+```ruby
+SemanticCache.configure do |c|
+  # Similarity threshold (0.0 to 1.0). Higher = stricter matching.
+  c.similarity_threshold = 0.85
+
+  # Embedding model
+  c.embedding_model = "text-embedding-3-small"
+
+  # OpenAI API key
+  c.openai_api_key = ENV["OPENAI_API_KEY"]
+
+  # Default TTL for cached entries (nil = no expiry)
+  c.default_ttl = 3600  # 1 hour
+
+  # Cache store: :memory or :redis
+  c.store = :memory
+  c.store_options = {}  # passed to Redis.new if store is :redis
+
+  # Cost tracking
+  c.track_costs = true
+end
+```
+
+## Cache Stores
+
+### In-Memory (default)
+
+Thread-safe, no dependencies. Good for development and single-process apps.
+
+```ruby
+cache = SemanticCache.new(store: :memory)
+```
+
+### Redis
+
+For production, multi-process, and distributed apps. Requires the `redis` gem.
+
+```ruby
+gem "redis"
+```
+
+```ruby
+cache = SemanticCache.new(
+  store: :redis,
+  store_options: { url: "redis://localhost:6379/0" }
+)
+```
+
+### Custom Store
+
+Any object that responds to `write`, `entries`, `delete`, `invalidate_by_tags`, `clear`, and `size`:
+
+```ruby
+cache = SemanticCache.new(store: MyCustomStore.new)
+```
+
+## TTL & Tag-Based Invalidation
+
+```ruby
+# TTL — auto-expires after 1 hour
+cache.fetch("Latest news?", ttl: 3600) do
+  fetch_news
+end
+
+# Tags — group related entries for bulk invalidation
+cache.fetch("Ruby version?", tags: [:ruby, :versions]) do
+  "3.3.0"
+end
+
+cache.fetch("Best framework?", tags: [:ruby, :frameworks]) do
+  "Rails"
+end
+
+# Invalidate all entries tagged :versions
+cache.invalidate(tags: [:versions])
+```
+
+## Multi-Model Support
+
+Convenience methods for different LLM providers:
+
+```ruby
+cache.fetch_openai("query", model: "gpt-4o") do
+  openai.chat(...)
+end
+
+cache.fetch_anthropic("query", model: "claude-sonnet-4-20250514") do
+  anthropic.messages(...)
+end
+
+cache.fetch_gemini("query", model: "gemini-pro") do
+  gemini.generate(...)
+end
+```
+
+## Client Wrapper
+
+Wrap an existing OpenAI client to cache all chat calls automatically:
+
+```ruby
+require "openai"
+
+client = OpenAI::Client.new(access_token: "sk-...")
+cached_client = SemanticCache.wrap(client)
+
+# All chat calls are now cached
+response = cached_client.chat(parameters: {
+  model: "gpt-4o",
+  messages: [{ role: "user", content: "What is Ruby?" }]
+})
+
+# Access cache stats
+cached_client.semantic_cache.current_stats
+
+# Other methods are delegated to the original client
+cached_client.models  # => calls client.models directly
+```
+
+## Cost Tracking & Stats
+
+```ruby
+cache = SemanticCache.new
+
+# After some usage...
+cache.current_stats
+# => {
+#   hits: 156,
+#   misses: 44,
+#   total_queries: 200,
+#   hit_rate: 78.0,
+#   savings: "$23.45",
+#   ...
+# }
+
+puts cache.detailed_stats
+# Total queries: 200
+# Cache hits: 156
+# Cache misses: 44
+# Hit rate: 78.0%
+# Total savings: $23.45
+
+puts cache.savings_report
+# Total saved: $23.45 (156 cached calls)
+```
+
+Custom model costs:
+
+```ruby
+SemanticCache.configure do |c|
+  c.model_costs["my-custom-model"] = { input: 0.01, output: 0.03 }
+end
+```
+
+## Rails Integration
+
+```ruby
+# Gemfile
+gem "semantic-cache"
+```
+
+```ruby
+# config/initializers/semantic_cache.rb
+require "semantic_cache/rails"
+
+SemanticCache.configure do |c|
+  c.openai_api_key = Rails.application.credentials.openai_api_key
+  c.store = :redis
+  c.store_options = { url: ENV["REDIS_URL"] }
+end
+```
+
+### Using the Concern
+
+```ruby
+class ChatController < ApplicationController
+  include SemanticCache::Cacheable
+
+  cache_ai_calls only: [:create], ttl: 1.hour
+
+  def create
+    response = SemanticCache.current.fetch(params[:message], model: "gpt-4o") do
+      openai_client.chat(parameters: {
+        model: "gpt-4o",
+        messages: [{ role: "user", content: params[:message] }]
+      })
+    end
+
+    render json: { response: response }
+  end
+end
+```
+
+### Per-User Namespacing
+
+```ruby
+class ApplicationController < ActionController::Base
+  around_action :with_semantic_cache
+
+  private
+
+  def with_semantic_cache
+    SemanticCache.with_cache(namespace: "user_#{current_user.id}") do
+      yield
+    end
+  end
+end
+```
+
+## Demo
+
+Run the built-in demo (no API key needed):
+
+```bash
+ruby examples/demo.rb --simulate
+```
+
+Or with a real API key:
+
+```bash
+OPENAI_API_KEY=sk-... ruby examples/demo.rb
+```
+
+## Development
+
+```bash
+bundle install
+bundle exec rspec
+```
+
+## License
+
+MIT License. See [LICENSE](LICENSE).

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/lib/semantic_cache/cache.rb

@@ -0,0 +1,187 @@
+# frozen_string_literal: true
+
+require "digest"
+
+module SemanticCache
+  class Cache
+    attr_reader :stats
+
+    def initialize(
+      similarity_threshold: nil,
+      embedding_model: nil,
+      store: nil,
+      store_options: {},
+      default_ttl: nil,
+      namespace: nil,
+      track_costs: nil
+    )
+      config = SemanticCache.configuration
+
+      @threshold = similarity_threshold || config.similarity_threshold
+      @default_ttl = default_ttl || config.default_ttl
+      @track_costs = track_costs.nil? ? config.track_costs : track_costs
+      @namespace = namespace || config.namespace
+
+      @embedding = Embedding.new(
+        model: embedding_model || config.embedding_model,
+        api_key: config.openai_api_key
+      )
+
+      @store = build_store(store || config.store, store_options.empty? ? config.store_options : store_options)
+      @stats = Stats.new
+    end
+
+    # Fetch a cached response or execute the block and cache the result.
+    #
+    #   cache.fetch("What is Ruby?") do
+    #     openai.chat(messages: [{ role: "user", content: "What is Ruby?" }])
+    #   end
+    #
+    # Options:
+    #   ttl:   - Time-to-live in seconds (overrides default)
+    #   tags:  - Array of tags for grouped invalidation
+    #   model: - Model name for cost tracking
+    def fetch(query, ttl: nil, tags: [], model: nil, metadata: {}, &block)
+      raise ArgumentError, "A block is required" unless block_given?
+
+      start_time = Time.now
+
+      # Generate embedding for the query
+      query_embedding = @embedding.generate(query)
+
+      # Search for a semantically similar cached entry
+      match = find_similar(query_embedding)
+
+      if match
+        elapsed = ((Time.now - start_time) * 1000).round(2)
+        saved_cost = estimate_cost(model)
+        @stats.record_hit(saved_cost: saved_cost, response_time: elapsed)
+        return match.response
+      end
+
+      # Cache miss — execute the block
+      response = block.call
+
+      elapsed = ((Time.now - start_time) * 1000).round(2)
+      @stats.record_miss(response_time: elapsed)
+
+      # Store the new entry
+      entry = Entry.new(
+        query: query,
+        embedding: query_embedding,
+        response: response,
+        model: model,
+        tags: Array(tags),
+        ttl: ttl || @default_ttl,
+        metadata: metadata
+      )
+
+      key = generate_key(query)
+      @store.write(key, entry)
+
+      response
+    end
+
+    # Convenience methods for specific providers
+
+    def fetch_openai(query, model: "gpt-4o", **options, &block)
+      fetch(query, model: model, **options, &block)
+    end
+
+    def fetch_anthropic(query, model: "claude-sonnet-4-20250514", **options, &block)
+      fetch(query, model: model, **options, &block)
+    end
+
+    def fetch_gemini(query, model: "gemini-pro", **options, &block)
+      fetch(query, model: model, **options, &block)
+    end
+
+    # Invalidate cached entries by tags.
+    #
+    #   cache.invalidate(tags: [:product_info])
+    #   cache.invalidate(tags: "user_data")
+    def invalidate(tags:)
+      @store.invalidate_by_tags(Array(tags))
+    end
+
+    # Clear all cached entries.
+    def clear
+      @store.clear
+      @stats.reset!
+    end
+
+    # Return current cache statistics as a Hash.
+    def current_stats
+      @stats.to_h
+    end
+
+    # Return a formatted stats report string.
+    def detailed_stats
+      @stats.report
+    end
+
+    # Savings report string.
+    def savings_report
+      s = @stats
+      "Total saved: #{format("$%.2f", s.total_savings)} (#{s.hits} cached calls)"
+    end
+
+    # Number of entries currently in the cache.
+    def size
+      @store.size
+    end
+
+    private
+
+    def find_similar(query_embedding)
+      entries = @store.entries
+      return nil if entries.empty?
+
+      best_match = nil
+      best_score = -1.0
+
+      entries.each do |entry|
+        next if entry.expired?
+
+        score = Similarity.cosine(query_embedding, entry.embedding)
+        if score > best_score
+          best_score = score
+          best_match = entry
+        end
+      end
+
+      return nil if best_match.nil? || best_score < @threshold
+
+      best_match
+    end
+
+    def estimate_cost(model)
+      return 0.0 unless @track_costs && model
+
+      costs = SemanticCache.configuration.cost_for(model)
+      # Rough estimate: average request ~500 input tokens, ~200 output tokens
+      ((costs[:input] * 0.5) + (costs[:output] * 0.2)).round(6)
+    end
+
+    def generate_key(query)
+      Digest::SHA256.hexdigest("#{@namespace}:#{query}")[0, 16]
+    end
+
+    def build_store(type, options)
+      case type
+      when :memory, "memory"
+        Stores::Memory.new(**options)
+      when :redis, "redis"
+        Stores::Redis.new(**options)
+      when Stores::Memory, Stores::Redis
+        type # Already instantiated
+      else
+        if type.respond_to?(:write) && type.respond_to?(:entries)
+          type # Duck-typed custom store
+        else
+          raise ConfigurationError, "Unknown store type: #{type}. Use :memory or :redis."
+        end
+      end
+    end
+  end
+end

data/lib/semantic_cache/client_wrapper.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+module SemanticCache
+  # Wraps an OpenAI client (or compatible) to automatically cache chat calls.
+  #
+  #   client = OpenAI::Client.new
+  #   cached = SemanticCache::ClientWrapper.new(client)
+  #
+  #   # All chat calls are automatically cached
+  #   cached.chat(parameters: { model: "gpt-4o", messages: [...] })
+  #
+  # Or use the shorthand:
+  #
+  #   cached = SemanticCache.wrap(client)
+  #
+  class ClientWrapper
+    def initialize(client, cache: nil, **cache_options)
+      @client = client
+      @cache = cache || Cache.new(**cache_options)
+    end
+
+    def chat(parameters: {}, **kwargs)
+      messages = parameters[:messages] || parameters["messages"] || []
+      model = parameters[:model] || parameters["model"]
+
+      # Use the last user message as the cache key
+      user_message = messages.reverse.find { |m| m[:role] == "user" || m["role"] == "user" }
+      query = user_message && (user_message[:content] || user_message["content"])
+
+      if query
+        @cache.fetch(query, model: model) do
+          @client.chat(parameters: parameters, **kwargs)
+        end
+      else
+        @client.chat(parameters: parameters, **kwargs)
+      end
+    end
+
+    # Delegate everything else to the wrapped client
+    def method_missing(method, ...)
+      if @client.respond_to?(method)
+        @client.send(method, ...)
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(method, include_private = false)
+      @client.respond_to?(method, include_private) || super
+    end
+
+    # Access the underlying cache for stats, invalidation, etc.
+    def semantic_cache
+      @cache
+    end
+  end
+
+  class << self
+    # Convenience: wrap a client with semantic caching.
+    #
+    #   cached_client = SemanticCache.wrap(openai_client)
+    def wrap(client, **options)
+      ClientWrapper.new(client, **options)
+    end
+  end
+end

data/lib/semantic_cache/configuration.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module SemanticCache
+  class Configuration
+    attr_accessor :similarity_threshold,
+                  :embedding_model,
+                  :openai_api_key,
+                  :default_ttl,
+                  :store,
+                  :store_options,
+                  :track_costs,
+                  :model_costs,
+                  :namespace
+
+    # Cost per 1K tokens (USD)
+    DEFAULT_MODEL_COSTS = {
+      # OpenAI
+      "gpt-4" => { input: 0.03, output: 0.06 },
+      "gpt-4-turbo" => { input: 0.01, output: 0.03 },
+      "gpt-4o" => { input: 0.005, output: 0.015 },
+      "gpt-4o-mini" => { input: 0.00015, output: 0.0006 },
+      "gpt-3.5-turbo" => { input: 0.0005, output: 0.0015 },
+      # Anthropic
+      "claude-sonnet-4-20250514" => { input: 0.003, output: 0.015 },
+      "claude-3-5-haiku-20241022" => { input: 0.001, output: 0.005 },
+      # Gemini
+      "gemini-pro" => { input: 0.0005, output: 0.0015 },
+      "gemini-1.5-pro" => { input: 0.00125, output: 0.005 },
+      # Embedding (cost per 1K tokens)
+      "text-embedding-3-small" => { input: 0.00002, output: 0.0 },
+      "text-embedding-3-large" => { input: 0.00013, output: 0.0 }
+    }.freeze
+
+    def initialize
+      @similarity_threshold = 0.85
+      @embedding_model = "text-embedding-3-small"
+      @openai_api_key = ENV["OPENAI_API_KEY"]
+      @default_ttl = nil # No expiry by default
+      @store = :memory
+      @store_options = {}
+      @track_costs = true
+      @model_costs = DEFAULT_MODEL_COSTS.dup
+      @namespace = "semantic_cache"
+    end
+
+    def cost_for(model)
+      model_costs[model] || { input: 0.001, output: 0.002 }
+    end
+  end
+end