jekyll_ai_related_posts 0.1.3 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bd10032e0e9758acac7f0dc26d79aaa2a8de36e8872be41ecd62f915b4422847
-  data.tar.gz: f999747d95b4b9dce7c1fd6fcad198959a1734c13dfc50cbdd57bdb267f0fcdc
+  metadata.gz: 4cdd84629ee4f629c87e6db6e716988d931626bcce1146209781a1ead8e7360c
+  data.tar.gz: 36ec471f3873c0eb6678c2c35c2b0d58da00104abb56f571f0922a31b7137ee0
 SHA512:
-  metadata.gz: ea9b61e8f8e3b9148146eece6a453abc57e720c295db1b99d11d6ef6786c206a4efdca194c6c0d8287133699517775b242aa9b4cae5af1cc10ee40903be3c0ab
-  data.tar.gz: 9d43069c05704e7c032840e4fc6409da6fd2ca437b5b2cf282021a84b0c64fa9b1761c03ab3694759261cf10b26cd087a49ace883f1d4c85e5f365d90d0b8bb2
+  metadata.gz: d35cec25d660ea48f8062240305850943710930446b121a349611cb4212b8d368c3164b0b9cc44549eb757f7278fe10b5a2091f19a2c9ff3212ed8b9ee7d3775
+  data.tar.gz: 0aa5878a2ada754441f1bc8e69e765407bfbaaea4f80a74c0951b1690c70f2a6e9c272bf8df6f86efbca3bea11e94067a723637fe577b997417dfb661bdf9a5d

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 ## [Unreleased]
 
+- feat: Generic LLM provider support (OpenAI-compatible APIs like OpenRouter) with configurable `api_url` and `model`. Embedding dimensions are auto-discovered from the API.
+
+## [0.1.4] - 2024-10-12
+
+- Better log messages (improved clarity about what's happening).
+
+## [0.1.3] - 2024-05-15
+
+- Better (nicer to read) log messages.
+
 ## [0.1.2] - 2024-04-24
 
 - [Bugfix] Include up to 10 related posts (like native Jekyll does) instead of just 3.

data/README.md

@@ -46,23 +46,40 @@ exclude:
 
 ## Configuration
 
-All config for this plugin sits under a top-level `ai_related_posts` key.
+All config for this plugin sits under a top-level `ai_related_posts` key in
+Jekyll's `_config.yml`.
 
-The only required config is `openai_api_key` -- we need to authenticate to the
+The only required config is an API key -- we need to authenticate to the
 API to fetch embedding vectors.
 
-- **openai_api_key** Your OpenAI API key, used to fetch embeddings.
+- **api_key** (or `openai_api_key` for backward compatibility) Your API key, used to fetch embeddings.
+- **api_url** (optional, default `https://api.openai.com`). The base URL for the embeddings API.
+- **model** (optional, default `text-embedding-3-small`). The model to use for embeddings.
 - **fetch_enabled** (optional, default `true`). If true, fetch embeddings. If
   false, don't fetch embeddings. If this is a string (like `prod`), fetch
   embeddings only when the `JEKYLL_ENV` environment variable is equal to the
   string. (This is useful if you want to reduce API costs by only fetching
   embeddings on production builds.)
 
-### Example Config
+**Important:** The plugin stores the model and dimensions in the cache database. If you change the model or dimensions in your config, the plugin will detect the mismatch and exit with an error. You can either:
+- Update your config to match the cached values, or
+- Delete the cache file (`.ai_related_posts_cache.sqlite3`) and it will be regenerated with the new model/dimensions.
+
+### Example Config: OpenAI (default)
+
+```yaml
+ai_related_posts:
+  api_key: sk-proj-abc123
+  fetch_enabled: prod
+```
+
+### Example Config: OpenRouter
 
 ```yaml
 ai_related_posts:
-  openai_api_key: sk-proj-abc123
+  api_key: sk-or-v1-abc123
+  api_url: https://openrouter.ai/api
+  model: openai/text-embedding-3-small
   fetch_enabled: prod
 ```
 
@@ -128,14 +145,17 @@ fees if done frequently).
 ## How It Works
 
 Jekyll AI Related Posts is implemented as a Jekyll Generator plugin. During the
-build process, the plugin will call the [OpenAI Embeddings
-API](https://platform.openai.com/docs/guides/embeddings) to fetch the vector
-embedding for a string containing the title, tags, and categories of your
-article. It's not necessary to use the full post text, in most cases the title
-and tags produce very accurate results because the LLM knows when topics are
-related even if they never use identical words. This is also why the LLM
-produces better results than LSI. These vector embeddings are cached in a SQLite
-database. To query for related posts, we query the cached vectors using the
+build process, the plugin will call an embeddings API (by default, the [OpenAI
+Embeddings API](https://platform.openai.com/docs/guides/embeddings)) to fetch
+the vector embedding for a string containing the title, tags, and categories of
+your article. The plugin works with any OpenAI-compatible embeddings API, such as
+[OpenRouter](https://openrouter.ai/).
+
+It's not necessary to use the full post text, in most cases the title and tags
+produce very accurate results because the LLM knows when topics are related even
+if they never use identical words. This is also why the LLM produces better
+results than LSI. These vector embeddings are cached in a SQLite database. To
+query for related posts, we query the cached vectors using the
 [sqlite-vss](https://github.com/asg017/sqlite-vss) plugin.
 
 ## Development

data/jekyll_ai_related_posts.gemspec

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+require_relative "lib/jekyll_ai_related_posts/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "jekyll_ai_related_posts"
+  spec.version = JekyllAiRelatedPosts::VERSION
+  spec.authors = [ "Mike Kasberg" ]
+  spec.email = [ "kasberg.mike@gmail.com" ]
+
+  spec.summary = "Populate ai_related_posts using Open AI embeddings"
+  spec.description = "Populate ai_related_posts using Open AI embeddings"
+  spec.homepage = "https://github.com/mkasberg/jekyll_ai_related_posts"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 3.0.0"
+
+  spec.metadata["allowed_push_host"] = "https://rubygems.org"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/mkasberg/jekyll_ai_related_posts"
+  spec.metadata["changelog_uri"] = "https://github.com/mkasberg/jekyll_ai_related_posts/blob/main/CHANGELOG.md"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) ||
+        f.start_with?(*%w[bin/ test/ spec/ features/ .git .github appveyor Gemfile])
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = [ "lib" ]
+
+  # Uncomment to register a new dependency of your gem
+  # spec.add_dependency "example-gem", "~> 1.0"
+  spec.add_dependency "activerecord", "~> 7.0"
+  spec.add_dependency "faraday", "~> 2.9"
+  spec.add_dependency "jekyll", ">= 3.0"
+  spec.add_dependency "sqlite3", "~> 1.4"
+  spec.add_dependency "sqlite-vss", "~> 0.1.2"
+  spec.add_dependency "zeitwerk", "~> 2.6"
+
+  spec.add_development_dependency "ostruct", "~> 0.6"
+
+  # For more information and examples about making a new gem, check out our
+  # guide at: https://bundler.io/guides/creating_gem.html
+end

data/lib/jekyll_ai_related_posts/api_embeddings.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+require "faraday"
+
+module JekyllAiRelatedPosts
+  class ApiEmbeddings
+    DEFAULT_API_URL = "https://api.openai.com"
+    DEFAULT_MODEL = "text-embedding-3-small"
+    DEFAULT_DIMENSIONS = 1536
+
+    def initialize(api_key, api_url: nil, model: nil, connection: nil)
+      @api_url = api_url || DEFAULT_API_URL
+      @model = model || DEFAULT_MODEL
+      @dimensions = nil
+
+      @connection = if connection.nil?
+        Faraday.new(url: @api_url) do |builder|
+          builder.request :authorization, "Bearer", api_key
+          builder.request :json
+          builder.response :json
+          builder.response :raise_error
+        end
+      else
+        connection
+      end
+    end
+
+    def dimensions
+      @dimensions ||= discover_dimensions
+    end
+
+    def model
+      @model
+    end
+
+    def embedding_for(text)
+      res = @connection.post("v1/embeddings") do |req|
+        req.body = {
+          input: text,
+          model: @model
+        }
+      end
+
+      data = res.body
+      unless data.is_a?(Hash) &&
+             data["data"].is_a?(Array) &&
+             data["data"][0].is_a?(Hash) &&
+             data["data"][0]["embedding"].is_a?(Array)
+        Jekyll.logger.error "AI Related Posts:", "Unexpected API response structure!"
+        Jekyll.logger.error "AI Related Posts:", "Response body: #{data.inspect}"
+        raise Error, "Unexpected API response: embedding data not found"
+      end
+
+      embedding = data["data"][0]["embedding"]
+      @dimensions ||= embedding.length
+      embedding
+    rescue Faraday::Error => e
+      Jekyll.logger.error "AI Related Posts:", "Error response from embeddings API!"
+      Jekyll.logger.error "AI Related Posts:", e.inspect
+
+      raise
+    end
+
+    private
+
+    def discover_dimensions
+      embedding_for("test").length
+    end
+  end
+end

data/lib/jekyll_ai_related_posts/generator.rb

@@ -9,9 +9,15 @@ require "json"
 module JekyllAiRelatedPosts
   class Generator < Jekyll::Generator
     def generate(site)
-      Jekyll.logger.info "AI Related Posts:", "Generating related posts..."
+      Jekyll.logger.debug "AI Related Posts:", "Generating related posts..."
 
       @site = site
+      @stats = {
+        cache_hits: 0,
+        cache_misses: 0
+      }
+      @embeddings_fetcher = new_fetcher if fetch_enabled?
+
       setup_database
 
       @indexed_posts = {}
@@ -20,7 +26,7 @@ module JekyllAiRelatedPosts
       end
 
       if fetch_enabled?
-        @embeddings_fetcher = new_fetcher
+        validate_cache_metadata
 
         @site.posts.docs.each do |p|
           ensure_embedding_cached(p)
@@ -29,13 +35,25 @@ module JekyllAiRelatedPosts
         @site.posts.docs.each do |p|
           find_related(p)
         end
+        Jekyll.logger.info "AI Related Posts:", "Found #{@stats[:cache_hits]} cached embeddings; fetched #{@stats[:cache_misses]}"
       else
         Jekyll.logger.info "AI Related Posts:", "Fetch disabled. Using cached related posts data."
 
         @site.posts.docs.each do |p|
           fallback_generate_related(p)
         end
+
+        case @stats[:cache_misses]
+        when 0
+          Jekyll.logger.info "AI Related Posts:", "Found #{@stats[:cache_hits]} cached embeddings; all embeddings cached"
+        when 1
+          Jekyll.logger.info "AI Related Posts:", "Found #{@stats[:cache_hits]} cached embeddings; skipped 1 fetch"
+        else
+          Jekyll.logger.info "AI Related Posts:", "Found #{@stats[:cache_hits]} cached embeddings; skipped #{@stats[:cache_misses]} fetches"
+        end
       end
+
+      Jekyll.logger.debug "AI Related Posts:", "Done generating related posts"
     end
 
     private
@@ -54,8 +72,14 @@ module JekyllAiRelatedPosts
     def fallback_generate_related(post)
       existing = Models::Post.find_by(relative_path: post.relative_path)
       if existing.nil?
+        @stats[:cache_misses] += 1
         post.data["ai_related_posts"] = post.related_posts
       else
+        if existing.embedding_text == embedding_text(post)
+          @stats[:cache_hits] += 1
+        else
+          @stats[:cache_misses] += 1
+        end
         find_related(post)
       end
     end
@@ -63,9 +87,15 @@ module JekyllAiRelatedPosts
     def new_fetcher
       case @site.config["ai_related_posts"]["embeddings_source"]
       when "mock"
-        MockEmbeddings.new
+        model = @site.config["ai_related_posts"]["model"]
+        dimensions = @site.config["ai_related_posts"]["dimensions"]
+        MockEmbeddings.new(model: model, dimensions: dimensions)
       else
-        OpenAiEmbeddings.new(@site.config["ai_related_posts"]["openai_api_key"])
+        api_key = @site.config["ai_related_posts"]["api_key"] ||
+          @site.config["ai_related_posts"]["openai_api_key"]
+        api_url = @site.config["ai_related_posts"]["api_url"]
+        model = @site.config["ai_related_posts"]["model"]
+        ApiEmbeddings.new(api_key, api_url: api_url, model: model)
       end
     end
 
@@ -81,20 +111,25 @@ module JekyllAiRelatedPosts
         existing = nil
       end
 
-      return unless existing.nil?
+      if existing.nil?
+        @stats[:cache_misses] += 1
 
-      Models::Post.create!(
-        relative_path: post.relative_path,
-        embedding_text: embedding_text(post),
-        embedding: embedding_for(post).to_json
-      )
+        Models::Post.create!(
+          relative_path: post.relative_path,
+          embedding_text: embedding_text(post),
+          embedding: embedding_for(post).to_json
+        )
 
-      sql = <<-SQL
-          INSERT INTO vss_posts (rowid, post_embedding)
-            SELECT rowid, embedding FROM posts WHERE relative_path = :relative_path;
-      SQL
-      ActiveRecord::Base.connection.execute(ActiveRecord::Base.sanitize_sql([ sql,
-                                                                             { relative_path: post.relative_path } ]))
+        sql = <<-SQL
+            INSERT INTO vss_posts (rowid, post_embedding)
+              SELECT rowid, embedding FROM posts WHERE relative_path = :relative_path;
+        SQL
+        ActiveRecord::Base.connection.execute(
+          ActiveRecord::Base.sanitize_sql([ sql, { relative_path: post.relative_path } ])
+        )
+      else
+        @stats[:cache_hits] += 1
+      end
     end
 
     def find_related(post)
@@ -108,9 +143,9 @@ module JekyllAiRelatedPosts
         LIMIT 10000;
       SQL
 
-      results = ActiveRecord::Base.connection.execute(ActiveRecord::Base.sanitize_sql([ sql, {
-                                                                                        relative_path: post.relative_path
-                                                                                      } ]))
+      results = ActiveRecord::Base.connection.execute(
+        ActiveRecord::Base.sanitize_sql([ sql, { relative_path: post.relative_path } ])
+      )
       # The first result is the post itself, with a distance of 0.
       rowids = results.sort_by { |r| r["distance"] }.drop(1).first(10).map { |r| r["rowid"] }
 
@@ -131,6 +166,14 @@ module JekyllAiRelatedPosts
       post.data["ai_related_posts"] = related_posts
     end
 
+    def dimensions
+      @embeddings_fetcher&.dimensions || ApiEmbeddings::DEFAULT_DIMENSIONS
+    end
+
+    def model
+      @embeddings_fetcher&.model || ApiEmbeddings::DEFAULT_MODEL
+    end
+
     def embedding_text(post)
       text = "Title: #{post.data["title"]}"
       text += "; Categories: #{post.data["categories"].join(", ")}" unless post.data["categories"].empty?
@@ -147,9 +190,16 @@ module JekyllAiRelatedPosts
     end
 
     def setup_database
+      db_path = @site.in_source_dir(".ai_related_posts_cache.sqlite3")
+      if File.exist?(db_path)
+        Jekyll.logger.debug "AI Related Posts:", "Found cache [.ai_related_posts_cache.sqlite3]"
+      else
+        Jekyll.logger.info "AI Related Posts:", "Creating cache [.ai_related_posts_cache.sqlite3]"
+      end
+
       ActiveRecord::Base.establish_connection(
         adapter: "sqlite3",
-        database: @site.in_source_dir(".ai_related_posts_cache.sqlite3")
+        database: db_path
       )
       # We don't need WAL mode for this.
       ActiveRecord::Base.connection.execute("PRAGMA journal_mode=DELETE;")
@@ -169,14 +219,51 @@ module JekyllAiRelatedPosts
       SQL
       ActiveRecord::Base.connection.execute(create_posts)
 
-      create_vss_posts = <<-SQL
-        CREATE VIRTUAL TABLE IF NOT EXISTS vss_posts using vss0(
-          post_embedding(#{OpenAiEmbeddings::DIMENSIONS})
+      unless table_exists?("vss_posts")
+        create_vss_posts = <<-SQL
+          CREATE VIRTUAL TABLE vss_posts using vss0(
+            post_embedding(#{dimensions})
+          );
+        SQL
+        ActiveRecord::Base.connection.execute(create_vss_posts)
+      end
+
+      create_cache_metadata = <<-SQL
+        CREATE TABLE IF NOT EXISTS cache_metadata(
+          key TEXT PRIMARY KEY,
+          value TEXT
         );
       SQL
-      ActiveRecord::Base.connection.execute(create_vss_posts)
+      ActiveRecord::Base.connection.execute(create_cache_metadata)
 
       Jekyll.logger.debug "AI Related Posts:", "DB setup complete"
     end
+
+    def table_exists?(name)
+      ActiveRecord::Base.connection.table_exists?(name)
+    end
+
+    def validate_cache_metadata
+      model = @embeddings_fetcher&.model || ApiEmbeddings::DEFAULT_MODEL
+
+      stored_model = ActiveRecord::Base.connection.execute(
+        "SELECT value FROM cache_metadata WHERE key = 'model';"
+      ).first
+
+      if stored_model && stored_model["value"] != model
+        Jekyll.logger.error "AI Related Posts:", "Cache model mismatch!"
+        Jekyll.logger.error "AI Related Posts:", "  Configured model: #{model}"
+        Jekyll.logger.error "AI Related Posts:", "  Cached model:     #{stored_model["value"]}"
+        Jekyll.logger.error "AI Related Posts:", "Either update your config to match the cached model, or delete the cache file (.ai_related_posts_cache.sqlite3) and it will be regenerated."
+        raise Error, "Cache model mismatch: configured=#{model}, cached=#{stored_model["value"]}"
+      end
+
+      # Store/update metadata if not present
+      if stored_model.nil?
+        ActiveRecord::Base.connection.execute(
+          ActiveRecord::Base.sanitize_sql([ "INSERT INTO cache_metadata (key, value) VALUES ('model', ?);", model ])
+        )
+      end
+    end
   end
 end

data/lib/jekyll_ai_related_posts/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module JekyllAiRelatedPosts
-  VERSION = "0.1.3"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jekyll_ai_related_posts
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.2.0
 platform: ruby
 authors:
 - Mike Kasberg
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-05-16 00:00:00.000000000 Z
+date: 2026-06-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -94,6 +94,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.6'
+- !ruby/object:Gem::Dependency
+  name: ostruct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
 description: Populate ai_related_posts using Open AI embeddings
 email:
 - kasberg.mike@gmail.com
@@ -109,10 +123,11 @@ files:
 - Rakefile
 - gemfiles/current.gemfile
 - gemfiles/jekyll3.gemfile
+- jekyll_ai_related_posts.gemspec
 - lib/jekyll_ai_related_posts.rb
+- lib/jekyll_ai_related_posts/api_embeddings.rb
 - lib/jekyll_ai_related_posts/generator.rb
 - lib/jekyll_ai_related_posts/models/post.rb
-- lib/jekyll_ai_related_posts/open_ai_embeddings.rb
 - lib/jekyll_ai_related_posts/version.rb
 homepage: https://github.com/mkasberg/jekyll_ai_related_posts
 licenses:
@@ -137,7 +152,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.6
+rubygems_version: 3.5.3
 signing_key:
 specification_version: 4
 summary: Populate ai_related_posts using Open AI embeddings

data/lib/jekyll_ai_related_posts/open_ai_embeddings.rb

@@ -1,38 +0,0 @@
-# frozen_string_literal: true
-
-require "faraday"
-
-module JekyllAiRelatedPosts
-  class OpenAiEmbeddings
-    DIMENSIONS = 1536
-
-    def initialize(api_key, connection: nil)
-      @connection = if connection.nil?
-                      Faraday.new(url: "https://api.openai.com") do |builder|
-                        builder.request :authorization, "Bearer", api_key
-                        builder.request :json
-                        builder.response :json
-                        builder.response :raise_error
-                      end
-      else
-                      connection
-      end
-    end
-
-    def embedding_for(text)
-      res = @connection.post("/v1/embeddings") do |req|
-        req.body = {
-          input: text,
-          model: "text-embedding-3-small"
-        }
-      end
-
-      res.body["data"].first["embedding"]
-    rescue Faraday::Error => e
-      Jekyll.logger.error "AI Related Posts:", "Error response from OpenAI API!"
-      Jekyll.logger.error "AI Related Posts:", e.inspect
-
-      raise
-    end
-  end
-end