embed_sync 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 42ea81557212045ba1955a65c47be73695e265fe5d807faa862598d76edcac2a
+  data.tar.gz: b50f6e052155f4e38620fb04a0e4cad96cd813e97bcd6def5e564604456ef911
+SHA512:
+  metadata.gz: 796b3a5fe7f8fd1370233162025afe79269a449d615f84f44e46ee63558a00cc1ca7b1e4d9796219d5389194a27f3bf3b3b066823200aeb1d2fff02070dd8380
+  data.tar.gz: 90fcc0a2dc2638d9cc0cf1045642e8920d9ac626c2ee5b890d771a978e3b5939288647435b5c1b357f825e79e2852840585fba081608c44d72ccf07b73fe4bbe

data/CHANGELOG.md

@@ -0,0 +1,18 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-10-30
+
+First release.
+
+- `has_embedding` macro: attribute list or lambda as the source, `model:`, `dimensions:`, `sync:`, `if:`/`unless:`
+- Change detection with a digest of the text, model and dimensions (`<column>_stale?`, `<column>_missing` scope)
+- Background sync after commit with `EmbedSync::EmbedJob`; `:inline` and disabled modes; `EmbedSync.without_sync`
+- Generators: `embed_sync:install` and `embed_sync MODEL attrs...` (pgvector migration with HNSW/IVFFlat index)
+- `EmbedSync.backfill` / `EmbedSync.stale_report` and the `embed_sync:backfill` / `embed_sync:stale` rake tasks
+- Providers: `:ruby_llm` (default) and `:fake`; custom providers are any object with `#embed(texts, model:)`
+- `Model.embed_query` and `Model.semantic_search` (neighbor), with an optional query cache
+- `embed.embed_sync` instrumentation and log subscriber; retries with backoff; Solid Queue concurrency limit
+- `EmbedSync::InvalidRequestError` for requests the provider rejects (text too long, bad key): not retried, and
+  backfill skips those records instead of stopping
+- Provider failures in `:inline` mode are logged instead of raising from an already-committed `save`
+- Embeddings computed for an older version of a record's text are discarded instead of overwriting newer ones

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Umair0092
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,187 @@
+# EmbedSync
+
+[![CI](https://github.com/Umair0092/embed_sync/actions/workflows/main.yml/badge.svg)](https://github.com/Umair0092/embed_sync/actions/workflows/main.yml)
+
+> Keep vector embeddings in sync with your ActiveRecord models.
+
+[neighbor](https://github.com/ankane/neighbor) lets you **search** vectors, and [ruby_llm](https://github.com/crmne/ruby_llm) lets you **generate** them. EmbedSync connects the two:
+
+- regenerates a record's embedding in a background job when the text it's built from changes
+- skips unchanged saves without calling the API again (it stores a digest of the text)
+- backfills existing tables in batches, safely re-runnable, with one API call per batch
+- re-embeds automatically when you switch embedding models
+- one-line semantic search
+
+```ruby
+class Product < ApplicationRecord
+  has_neighbors :embedding
+  has_embedding :embedding, from: %i[name description]
+end
+
+Product.create!(name: "Trail runner", description: "Waterproof running shoe")  # embedded in the background
+Product.semantic_search("shoes for rainy hikes", limit: 5)
+```
+
+## Requirements
+
+- Ruby 3.2+, Rails 7.1+ (tested on 7.1, 7.2, 8.0 and 8.1)
+- PostgreSQL with [pgvector](https://github.com/pgvector/pgvector), plus the `neighbor` gem for the `vector` column type and search
+- An embedding provider: [ruby_llm](https://github.com/crmne/ruby_llm) (default) or your own
+
+## Installation
+
+```ruby
+# Gemfile
+gem "embed_sync"
+gem "neighbor"
+gem "ruby_llm"
+```
+
+```bash
+bundle install
+bin/rails g embed_sync:install                    # config/initializers/embed_sync.rb
+bin/rails g embed_sync Product name description   # migration + has_embedding in app/models/product.rb
+bin/rails db:migrate
+bin/rails embed_sync:backfill                     # embed existing rows
+```
+
+Configure your API key for ruby_llm:
+
+```ruby
+# config/initializers/ruby_llm.rb
+RubyLLM.configure { |c| c.openai_api_key = Rails.application.credentials.openai_api_key }
+```
+
+## Usage
+
+### Declaring embeddings
+
+```ruby
+class Product < ApplicationRecord
+  belongs_to :category
+
+  has_neighbors :embedding
+
+  # Attribute names are joined with newlines (blank values are skipped)
+  has_embedding :embedding, from: %i[name description]
+
+  # ...or build the text yourself; the lambda runs on the record (or takes it: ->(product) { ... })
+  has_embedding :embedding, from: -> { "#{name}\nCategory: #{category&.name}" }
+end
+```
+
+| Option | Default | |
+|---|---|---|
+| `from:` | (required) | attribute, array of attributes, or lambda |
+| `model:` | `config.default_model` | embedding model name |
+| `dimensions:` | `provider_options[:dimensions]` | rejects vectors of a different size (`DimensionMismatchError`); not sent to the provider |
+| `sync:` | `config.sync` (`:async`) | `:async` (Active Job after commit), `:inline`, or `false` |
+| `if:` / `unless:` | none | only sync matching records, as with Rails callbacks |
+| `digest_column:` | `<column>_digest` | where the text digest is stored |
+
+A model can have several embeddings (`has_embedding :title_embedding, ...`). Pass `column:` to the methods below when it does.
+
+### Methods
+
+```ruby
+product.embedding_stale?          # text, model or dimensions changed since the last embedding?
+product.embedding_source_text     # the text that gets embedded
+product.refresh_embedding!        # re-embed now, even if fresh
+
+Product.embedding_missing         # scope: never embedded
+Product.embed_query("red shoes")  # vector for a search string, using the column's model
+Product.where(published: true).semantic_search("red shoes", limit: 10, distance: "cosine")
+# results respond to #neighbor_distance
+
+EmbedSync.without_sync { import_products! }        # no jobs during bulk imports (also inside a transaction)
+EmbedSync.backfill(Product.includes(:category))    # => number of records embedded
+EmbedSync.stale_report(Product)                    # => { total:, missing:, stale: }
+```
+
+### Rake tasks
+
+```bash
+bin/rails embed_sync:stale                          # all models with has_embedding
+bin/rails "embed_sync:backfill[Product]"
+bin/rails "embed_sync:backfill[Product,embedding]"
+```
+
+Backfill only embeds stale records, so you can stop it and run it again at any time.
+
+## Configuration
+
+```ruby
+EmbedSync.configure do |config|
+  config.provider = :ruby_llm                      # or :fake, or any object with #embed(texts, model:)
+  config.provider_options = { dimensions: 1536 }   # passed to RubyLLM.embed
+  config.default_model = "text-embedding-3-small"
+  config.sync = :async                             # :async, :inline or false
+  config.queue = :default
+  config.priority = nil
+  config.batch_size = 100                          # texts per API call when backfilling
+  config.concurrency_limit = nil                   # Solid Queue: max EmbedJobs running at once
+  config.query_cache = nil                         # e.g. Rails.cache, to cache search-query vectors
+  config.query_cache_expires_in = 86_400
+end
+```
+
+`:async` needs Active Job (apps created with `rails new --minimal` turn it off; use `sync: :inline` there).
+
+### Custom providers
+
+```ruby
+class MyProvider
+  # Return one vector (Array of Floats) per text, in order.
+  def embed(texts, model:)
+    MyClient.embeddings(input: texts, model: model).map { _1["embedding"] }
+  end
+end
+
+EmbedSync.config.provider = MyProvider.new
+```
+
+Raise `EmbedSync::ProviderError` for errors worth retrying (rate limits, timeouts) and `EmbedSync::InvalidRequestError` for requests that will never succeed (text too long, bad API key).
+
+## Production notes
+
+- **Retries:** `EmbedSync::EmbedJob` retries `ProviderError` 5 times with polynomial backoff, and discards jobs for deleted records and for `InvalidRequestError` (retrying a rejected request doesn't help).
+- **Failures don't break saves or backfills:** with `sync: :inline`, a provider error is logged and the record stays stale instead of raising from `save`. Backfill skips records the provider rejects and carries on; `embed_sync:stale` still lists them.
+- **Long texts:** embedding models have an input limit (about 8k tokens for OpenAI and nomic-embed-text). Records over it are rejected and stay stale; trim them in the source, e.g. `from: -> { "#{name}\n#{description}".first(20_000) }`.
+- **Concurrent edits:** if a record changes while its embedding is being generated, the older result is discarded rather than written over the newer text. Override in an initializer: `EmbedSync::EmbedJob.retry_on EmbedSync::ProviderError, wait: 1.minute, attempts: 10`.
+- **Rate limits:** with Solid Queue, `config.concurrency_limit = 2` caps how many embedding jobs run at once.
+- **Observability:** every provider call publishes `embed.embed_sync` (`record_class`, `column`, `model`, `count`) and is logged:
+  ```
+  EmbedSync (182.4ms) embedded 100 Product#embedding texts with text-embedding-3-small
+  ```
+- **Changing models:** the digest covers the model and dimensions (`has_embedding dimensions:` or `provider_options[:dimensions]`), so changing either marks every row stale and gives search queries a new cache key. For live search, add a second column (`--column=embedding_v2`), backfill it, switch `semantic_search(..., column: :embedding_v2)`, then drop the old one.
+
+## Testing your app
+
+Use the deterministic, offline `:fake` provider in tests. The install generator sets this up, with syncing off so tests don't embed unless they ask to:
+
+```ruby
+# config/initializers/embed_sync.rb
+config.provider = Rails.env.test? ? :fake : :ruby_llm
+config.sync = Rails.env.test? ? false : :async   # use :inline in tests that need real vectors
+```
+
+`EmbedSync.provider.calls` records every call made to the fake.
+
+## Development
+
+```bash
+bin/setup
+bundle exec rake                       # specs + RuboCop
+
+# Another Rails version
+BUNDLE_GEMFILE=gemfiles/rails_7.1.gemfile bundle install
+BUNDLE_GEMFILE=gemfiles/rails_7.1.gemfile bundle exec rspec
+
+# Integration specs against PostgreSQL + pgvector
+docker run -d -p 54329:5432 -e POSTGRES_PASSWORD=postgres pgvector/pgvector:pg17
+DATABASE_URL=postgres://postgres:postgres@localhost:54329/postgres bundle exec rspec spec/integration
+```
+
+## License
+
+MIT, see [LICENSE.txt](LICENSE.txt).

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/lib/embed_sync/backfill.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module EmbedSync
+  # Batch operations over whole tables: embed what's missing or out of date, and report on it.
+  module Backfill
+    module_function
+
+    # Embeds every stale record in batches, one provider call per batch. Safe to re-run: records
+    # embedded by an earlier (possibly interrupted) run are fresh and get skipped. Records the provider
+    # rejects (e.g. text too long) are logged and left stale instead of stopping the backfill.
+    # Accepts a model or a relation, e.g. Product.includes(:category). Yields the running total.
+    def run(scope, column: nil, batch_size: nil, force: false)
+      relation = scope.all
+      definition = relation.klass.embed_sync_definition(column)
+      written = 0
+      relation.find_in_batches(batch_size: batch_size || EmbedSync.config.batch_size) do |batch|
+        written += Sync.call(batch, definition.column, force: force, skip_invalid: true)
+        yield written if block_given?
+      end
+      written
+    end
+
+    # Returns { total:, missing:, stale: }. "missing" rows were never embedded; "stale" includes
+    # them plus rows whose text changed since. Loads every record, so it can take a while.
+    def report(scope, column: nil)
+      relation = scope.all
+      definition = relation.klass.embed_sync_definition(column)
+      stale = 0
+      relation.find_each { |record| stale += 1 if definition.stale?(record) }
+      { total: relation.count, missing: relation.where(definition.digest_column => nil).count, stale: stale }
+    end
+  end
+end

data/lib/embed_sync/configuration.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+module EmbedSync
+  # Global settings, changed through EmbedSync.configure.
+  class Configuration
+    attr_accessor :default_model, :queue, :priority, :batch_size, :query_cache, :query_cache_expires_in,
+                  :concurrency_limit
+    attr_reader :sync, :provider, :provider_options
+
+    def initialize
+      @provider = :ruby_llm
+      @provider_options = {}
+      @default_model = nil
+      @queue = :default
+      @batch_size = 100
+      @sync = :async
+      @query_cache = nil
+      @query_cache_expires_in = 86_400
+      @priority = nil
+      @concurrency_limit = nil
+    end
+
+    SYNC_MODES = [:async, :inline, false].freeze
+
+    # :async enqueues EmbedSync::EmbedJob after commit, :inline embeds right away, false turns syncing off.
+    def sync=(mode)
+      raise ConfigurationError, "sync must be one of #{SYNC_MODES.inspect}" unless SYNC_MODES.include?(mode)
+
+      @sync = mode
+    end
+
+    # Accepts a registered provider name (e.g. :fake) or any object that responds to #embed.
+    def provider=(value)
+      @provider = value
+      @resolved_provider = nil
+    end
+
+    def provider_options=(options)
+      @provider_options = (options || {}).to_h.transform_keys(&:to_sym)
+      @resolved_provider = nil
+    end
+
+    def resolved_provider
+      @resolved_provider ||= build_provider
+    end
+
+    private
+
+    def build_provider
+      case provider
+      when nil
+        raise ConfigurationError, "No embedding provider configured. Set EmbedSync.config.provider."
+      when Symbol, String
+        Providers.build(provider, **provider_options)
+      else
+        raise ConfigurationError, "#{provider.inspect} does not respond to #embed" unless provider.respond_to?(:embed)
+
+        provider
+      end
+    end
+  end
+end

data/lib/embed_sync/definition.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+require "digest"
+
+module EmbedSync
+  # Describes one embedding declared with has_embedding: which column holds the vector,
+  # where its text comes from, and which column stores the digest of that text.
+  class Definition
+    attr_reader :column, :source, :digest_column, :owner
+
+    def initialize(column:, from:, model: nil, digest_column: nil, sync: nil, dimensions: nil, owner: nil)
+      @owner = owner
+      @column = column.to_sym
+      @source = from
+      @model = model
+      @dimensions = dimensions
+      @digest_column = (digest_column || "#{column}_digest").to_sym
+      @sync = sync
+      validate_source!
+      validate_sync!
+    end
+
+    def sync_mode
+      @sync.nil? ? EmbedSync.config.sync : @sync
+    end
+
+    # has_embedding's dimensions:, else the size requested from the provider (provider_options[:dimensions]).
+    def dimensions
+      @dimensions || EmbedSync.config.provider_options[:dimensions]
+    end
+
+    def model_name
+      @model || EmbedSync.config.default_model ||
+        raise(ConfigurationError,
+              "No embedding model for #{column}. Pass model: or set EmbedSync.config.default_model.")
+    end
+
+    def source_text(record)
+      text = case source
+             when Proc then source.arity.zero? ? record.instance_exec(&source) : record.instance_exec(record, &source)
+             when Array then source.filter_map { |attr| record.public_send(attr).presence }.join("\n")
+             else record.public_send(source)
+             end
+      text.to_s
+    end
+
+    # Fingerprint of everything that shapes the vector: the text, the model and the dimensions.
+    # Changing the model or the dimensions therefore marks every record stale.
+    def digest(text)
+      Digest::SHA256.hexdigest("#{model_name}\n#{dimensions}\n#{text}")
+    end
+
+    def stale?(record)
+      text = source_text(record)
+      return true if record[digest_column] != digest(text)
+
+      text.present? && record[column].nil?
+    end
+
+    private
+
+    def validate_source!
+      return if source.is_a?(Proc) || source.is_a?(Symbol) || source.is_a?(String)
+      return if source.is_a?(Array) && source.any?
+
+      raise ArgumentError, "has_embedding from: must be an attribute name, an array of names or a proc"
+    end
+
+    def validate_sync!
+      return if @sync.nil? || Configuration::SYNC_MODES.include?(@sync)
+
+      raise ArgumentError, "has_embedding sync: must be one of #{Configuration::SYNC_MODES.inspect}"
+    end
+  end
+end

data/lib/embed_sync/embed_job.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+require "active_job"
+
+module EmbedSync
+  # Regenerates one record's embedding in the background.
+  class EmbedJob < ActiveJob::Base
+    queue_as { EmbedSync.config.queue }
+    queue_with_priority { EmbedSync.config.priority }
+
+    # Rate limits, timeouts and provider outages: back off and try again (3s, 18s, 83s, 258s).
+    retry_on ProviderError, wait: :polynomially_longer, attempts: 5
+
+    # The record was deleted before the job ran; nothing to embed.
+    discard_on ActiveJob::DeserializationError
+
+    # The provider rejected the text (too long, bad key, unknown model): retrying gives the same answer.
+    # The failure is logged and the record stays stale, so embed_sync:stale still reports it.
+    discard_on InvalidRequestError
+
+    def perform(record, column)
+      Sync.call([record], column, recheck: true)
+    end
+
+    # With Solid Queue, caps how many EmbedJobs run at once across all workers, so a large import can't
+    # exceed the provider's rate limit. Called at boot when config.concurrency_limit is set.
+    def self.apply_concurrency_limit(limit)
+      unless respond_to?(:limits_concurrency)
+        raise ConfigurationError, "config.concurrency_limit needs Solid Queue (limits_concurrency)"
+      end
+
+      limits_concurrency(to: limit, key: ->(*) { "embed_sync" }, duration: ActiveSupport::Duration.minutes(15))
+    end
+  end
+end

data/lib/embed_sync/errors.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module EmbedSync
+  class Error < StandardError; end
+
+  # Raised when EmbedSync is set up incorrectly (unknown provider, missing option, ...).
+  class ConfigurationError < Error; end
+
+  # Raised when an embedding provider fails or returns an unexpected response. EmbedJob retries these.
+  class ProviderError < Error; end
+
+  # Raised when the provider rejects the request itself (text too long, bad key, unknown model).
+  # Retrying won't help, so EmbedJob discards these and backfill skips the record.
+  class InvalidRequestError < Error; end
+
+  # Raised when vectors don't match has_embedding's dimensions:. Not retried: the setup has to change.
+  class DimensionMismatchError < Error; end
+end

data/lib/embed_sync/log_subscriber.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "active_support/log_subscriber"
+
+module EmbedSync
+  # Logs each provider call, e.g.
+  #   EmbedSync (182.4ms) embedded 100 Product#embedding texts with text-embedding-3-small
+  class LogSubscriber < ActiveSupport::LogSubscriber
+    def embed(event)
+      payload = event.payload
+      prefix = "EmbedSync (#{event.duration.round(1)}ms)"
+      target = [payload[:record_class], payload[:column]].compact.join("#")
+      if payload[:exception]
+        error { "#{prefix} failed to embed #{target}: #{payload[:exception].join(": ")}" }
+      else
+        info { "#{prefix} embedded #{payload[:count]} #{target} texts with #{payload[:model]}" }
+      end
+    end
+  end
+end
+
+EmbedSync::LogSubscriber.attach_to :embed_sync

data/lib/embed_sync/model.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+module EmbedSync
+  # Adds the has_embedding macro to ActiveRecord models.
+  module Model
+    extend ActiveSupport::Concern
+
+    CALLBACK_OPTIONS = %i[if unless].freeze
+    OPTIONS = (CALLBACK_OPTIONS + %i[model dimensions digest_column sync]).freeze
+
+    included do
+      class_attribute :embed_sync_definitions, instance_writer: false, default: {}
+    end
+
+    # True when the last save happened inside EmbedSync.without_sync. Captured at save time because
+    # after_commit runs later, after a surrounding transaction (and the without_sync block) has ended.
+    def embed_sync_skipped?
+      @embed_sync_skipped == true
+    end
+
+    private
+
+    def embed_sync_capture_sync_state
+      @embed_sync_skipped = !EmbedSync.sync_enabled?
+    end
+
+    # Class methods added to every model (Concern extends the model with ClassMethods).
+    module ClassMethods
+      # has_embedding :embedding, from: %i[name description]
+      # has_embedding :embedding, from: -> { "#{name}\n#{category&.name}" }, model: "text-embedding-3-small",
+      #               dimensions: 1536, sync: :inline, if: :published?
+      #
+      # Options: model:, dimensions:, digest_column:, sync: (:async, :inline, false), if:, unless:
+      def has_embedding(column = :embedding, from:, **options) # rubocop:disable Naming/PredicatePrefix
+        unknown = options.keys - OPTIONS
+        raise ArgumentError, "Unknown has_embedding options: #{unknown.join(", ")}" if unknown.any?
+
+        callback_options = options.slice(*CALLBACK_OPTIONS)
+        definition = Definition.new(column: column, from: from, owner: self, **options.except(*CALLBACK_OPTIONS))
+        self.embed_sync_definitions = embed_sync_definitions.merge(definition.column => definition)
+        define_embedding_methods(definition)
+        after_save :embed_sync_capture_sync_state
+        after_commit(on: %i[create update], **callback_options) { EmbedSync.sync_record(self, definition.column) }
+
+        definition
+      end
+
+      # Looks up a has_embedding by column; the column can be omitted when the model has only one.
+      def embed_sync_definition(column = nil)
+        if column
+          return embed_sync_definitions.fetch(column.to_sym) do
+            raise ArgumentError, "#{name} has no has_embedding for #{column.inspect}"
+          end
+        end
+
+        definitions = embed_sync_definitions.values
+        return definitions.first if definitions.one?
+
+        raise ArgumentError, "#{name} has #{definitions.size} embeddings; pass column: " \
+                             "(one of #{definitions.map(&:column).join(", ")})"
+      end
+
+      # Embeds a search query with the same model as the column, so it can be compared to stored vectors.
+      def embed_query(text, column: nil)
+        EmbedSync::Query.embed(text, embed_sync_definition(column))
+      end
+
+      # Product.semantic_search("red running shoes", limit: 5)
+      # Product.where(published: true).semantic_search("waterproof", column: :embedding, distance: "euclidean")
+      # Needs the neighbor gem (has_neighbors :embedding). Records respond to #neighbor_distance.
+      def semantic_search(query, column: nil, limit: 10, distance: "cosine")
+        definition = embed_sync_definition(column)
+        unless respond_to?(:nearest_neighbors)
+          raise ConfigurationError, "semantic_search needs the neighbor gem. Add gem \"neighbor\" and " \
+                                    "has_neighbors :#{definition.column} to #{name}."
+        end
+
+        nearest_neighbors(definition.column, embed_query(query, column: definition.column), distance: distance)
+          .limit(limit)
+      end
+
+      private
+
+      def define_embedding_methods(definition)
+        column = definition.column
+        define_method(:"#{column}_source_text") { definition.source_text(self) }
+        define_method(:"#{column}_stale?") { definition.stale?(self) }
+        define_method(:"refresh_#{column}!") { Sync.call([self], column, force: true) }
+        scope :"#{column}_missing", -> { where(definition.digest_column => nil) }
+      end
+    end
+  end
+end

data/lib/embed_sync/providers/base.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module EmbedSync
+  module Providers
+    # Interface every provider implements.
+    class Base
+      # Returns one vector (Array of Floats) per text, in the same order as +texts+.
+      def embed(texts, model:)
+        raise NotImplementedError, "#{self.class} must implement #embed(texts, model:)"
+      end
+    end
+  end
+end

data/lib/embed_sync/providers/fake.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require "digest"
+
+module EmbedSync
+  module Providers
+    # Deterministic, offline provider for tests: the same text and model always give the same
+    # unit-length vector. Records every call in #calls so tests can assert on API usage.
+    class Fake < Base
+      attr_reader :dimensions, :calls
+
+      def initialize(dimensions: 8)
+        super()
+        @dimensions = dimensions
+        @calls = []
+      end
+
+      def embed(texts, model:)
+        @calls << { texts: texts.dup, model: model }
+        texts.map { |text| vector_for(text, model) }
+      end
+
+      private
+
+      def vector_for(text, model)
+        bytes = Digest::SHA256.digest("#{model}\n#{text}").bytes
+        raw = Array.new(dimensions) { |i| (bytes[i % bytes.size] / 127.5) - 1.0 }
+        norm = Math.sqrt(raw.sum { |x| x * x })
+        norm.zero? ? raw : raw.map { |x| x / norm }
+      end
+    end
+  end
+end