rails-paradedb 0.9.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 25216744f6402f3ffdc06d8381f8ed801acec379dc15c71b92187b9e28a61b6d
-  data.tar.gz: 64dfcdcefa05ff52d0cd6b4b0292c005a0f4c2cf3a193ee07ac27bdae175ddd0
+  metadata.gz: 61970bcefe75f0514836a81ab38dd8e39fe1c6f311d55f35366120d2285325b9
+  data.tar.gz: 6179c3b6d0814dcaae1179977548c4b47e6f6447110596293f242d668727d79a
 SHA512:
-  metadata.gz: f3fb68ca97ffc5a30f6d503f15e97b02bc8db1de5aaeab020991848f777bc401fc62d06049490790ef75416b80c6a226edeb8cfe91e2bd8d8c3d76d12e95accb
-  data.tar.gz: 6c1e58314734c06d138a32f8a6928d13c7178e7db44e8460a45efbfb128e168d81f22dac53f74bc57ccd19f88f4ad0fdf60c3590dfcfd8d04e39b80d39faf531
+  metadata.gz: 38471cccf3d392cd481dbb22cde9f35c37f09a555c4e6081c5484d9c5ad1920082172276447a2401a22c0dd46bcd1a0846f2dee26678f556f913373996a91da0
+  data.tar.gz: 6176c6252901db66d8f8dd72972f8053fc6a187d0d10391705dde18fabe16a4992d60655a351b3af8643e9ca4a57c2baff58a11a2dd93cb63a708750785034c6

data/CHANGELOG.md

@@ -2,7 +2,27 @@
 
 All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.11.0] - 2026-08-04
+
+### Added
+
+- Vector index build options `centroid_ratio`, `training_samples_per_centroid`, and `cluster_replication` in `ParadeDB::Index` `index_options` and `add_paradedb_index` (pg_search 0.25.0+). They are emitted in the `WITH (...)` clause and round-tripped through the schema dumper.
+
+[0.10.0] - 2026-08-04
+
+### Added
+
+- Native pgvector `vector(n)` column support: ActiveRecord attribute type, `t.vector` / `add_column :table, :col, :vector, limit: n` migration DSL, and `schema.rb` dumping — no `neighbor` gem required.
+- Vector fields in ParadeDB indexes via `embedding: { metric: :l2 | :cosine | :ip }` in `ParadeDB::Index` fields and `add_paradedb_index`, emitted as `vector_l2_ops` / `vector_cosine_ops` / `vector_ip_ops` opclasses and round-tripped through the schema dumper.
+- `Model.nearest(column, vector, metric: nil)` for Top-K vector search. Orders by the pgvector distance operator (`<->`, `<=>`, `<#>`), defaults the metric from the index definition, and adds `key_field @@@ pdb.all()` when the relation has no ParadeDB predicate.
+- `l2_distance` / `cosine_distance` / `inner_product` / `vector_distance` Arel builder methods and `pdb_l2_distance` / `pdb_cosine_distance` / `pdb_inner_product` / `pdb_vector_distance` attribute predications.
+- `examples/vector_search` example. `examples/hybrid_rrf` now uses the native vector support instead of `neighbor`, and `examples/rag` retrieval combines full-text search with `nearest`.
+
+### Changed
+
+- **BREAKING**: Renamed the `bm25`-named migration helpers to `paradedb`: `add_bm25_index` is now `add_paradedb_index`, `remove_bm25_index` is now `remove_paradedb_index`, and `reindex_bm25` is now `reindex_paradedb_index`. The old names were removed; update migrations and `schema.rb` files to the new names.
+- **BREAKING**: Index creation always emits `USING paradedb`, which requires pg_search 0.25.0+. There is no option to select the legacy `bm25` access method.
+- **BREAKING**: The default index name is now `<table>_search_idx` (previously `<table>_bm25_idx`), and the index generator emits `Create<Model>SearchIndex` migrations named `create_<table>_search_index.rb`.
 
 [0.9.0] - 2026-07-14
 
@@ -154,7 +174,8 @@ All notable changes to this project will be documented in this file. The format
 - Schema dump/load round-trip for tokenizer configuration and index options
   (including `target_segment_count`)
 
-[Unreleased]: https://github.com/paradedb/rails-paradedb/compare/v0.9.0...HEAD
+[0.11.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.11.0
+[0.10.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.10.0
 [0.9.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.9.0
 [0.8.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.8.0
 [0.7.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.7.0

data/README.md

@@ -36,20 +36,26 @@
 
 ## ParadeDB for Rails
 
-The official ActiveRecord integration for [ParadeDB](https://paradedb.com) (powered by the [`pg_search`](https://github.com/paradedb/paradedb) Postgres extension), including first-class support for managing BM25 indexes and running queries using the full ParadeDB API. Follow the [getting started guide](https://docs.paradedb.com/documentation/getting-started/environment#rails) to begin.
+The official ActiveRecord integration for [ParadeDB](https://paradedb.com) (powered by the [`pg_search`](https://github.com/paradedb/paradedb) Postgres extension), including first-class support for managing ParadeDB indexes and running queries using the full ParadeDB API. Follow the [getting started guide](https://docs.paradedb.com/documentation/getting-started/environment#rails) to begin.
 
 ## Requirements & Compatibility
 
-| Component  | Supported                                        |
-| ---------- | ------------------------------------------------ |
-| Ruby       | 3.2+                                             |
-| Rails      | 7.2+                                             |
-| ParadeDB   | 0.22.0+                                          |
-| PostgreSQL | 15+ (PostgreSQL adapter with ParadeDB extension) |
+| Component  | Supported                                                         |
+| ---------- | ----------------------------------------------------------------- |
+| Ruby       | 3.2+                                                              |
+| Rails      | 7.2+                                                              |
+| ParadeDB   | 0.25.0+                                                           |
+| PostgreSQL | 15+ (PostgreSQL adapter with ParadeDB extension)                  |
+| pgvector   | Required for vector search; included in the ParadeDB Docker image |
+
+## Vector Search
+
+rails-paradedb supports full-text search and vector search over pgvector `vector(n)` columns. See the [vector search documentation](https://docs.paradedb.com/documentation/vector/overview) for details.
 
 ## Examples
 
 - [Quickstart](examples/quickstart/quickstart.rb)
+- [Vector Search](examples/vector_search/vector_search.rb)
 - [Faceted Search](examples/faceted_search/faceted_search.rb)
 - [Autocomplete](examples/autocomplete/autocomplete.rb)
 - [More Like This](examples/more_like_this/more_like_this.rb)

data/lib/generators/parade_db/index/index_generator.rb

@@ -15,7 +15,7 @@ module ParadeDB
       class_option :concurrent, type: :boolean, default: false,
                                 desc: "Add disable_ddl_transaction! to the migration (required for concurrent index creation)"
 
-      desc "Creates a ParadeDB::Index class and a BM25 index migration for MODEL."
+      desc "Creates a ParadeDB::Index class and a ParadeDB index migration for MODEL."
 
       def self.next_migration_number(dirname)
         ActiveRecord::Generators::Base.next_migration_number(dirname)
@@ -26,17 +26,17 @@ module ParadeDB
       end
 
       def create_migration_file
-        migration_template "migration.rb.tt", "db/migrate/create_#{table_name}_bm25_index.rb"
+        migration_template "migration.rb.tt", "db/migrate/create_#{table_name}_search_index.rb"
       end
 
       private
 
       def index_name
-        "#{table_name}_bm25_idx"
+        "#{table_name}_search_idx"
       end
 
       def migration_class_name
-        "Create#{class_name}Bm25Index"
+        "Create#{class_name}SearchIndex"
       end
     end
   end

data/lib/generators/parade_db/index/templates/migration.rb.tt

@@ -10,6 +10,6 @@ class <%= migration_class_name %> < ActiveRecord::Migration[<%= ActiveRecord::Mi
   end
 
   def down
-    remove_bm25_index :<%= table_name %>, name: :<%= index_name %>, if_exists: true
+    remove_paradedb_index :<%= table_name %>, name: :<%= index_name %>, if_exists: true
   end
 end

data/lib/parade_db/arel/builder.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 require "date"
 require_relative "../tokenizer_sql"
+require_relative "../vector"
 
 module ParadeDB
   module Arel
@@ -220,6 +221,23 @@ module ParadeDB
         infix("@@@", column_node(column), rhs)
       end
 
+      def l2_distance(column, vector)
+        vector_distance(column, vector, metric: :l2)
+      end
+
+      def cosine_distance(column, vector)
+        vector_distance(column, vector, metric: :cosine)
+      end
+
+      def inner_product(column, vector)
+        vector_distance(column, vector, metric: :ip)
+      end
+
+      def vector_distance(column, vector, metric: ParadeDB::Vector::DEFAULT_METRIC)
+        operator = ParadeDB::Vector::DISTANCE_OPERATORS.fetch(ParadeDB::Vector.normalize_metric(metric))
+        infix(operator, column_node(column), vector_operand(vector))
+      end
+
       def score(key)
         ::Arel::Nodes::NamedFunction.new("pdb.score", [column_node(key)])
       end
@@ -338,6 +356,13 @@ module ParadeDB
         ::Arel::Nodes.build_quoted(value)
       end
 
+      def vector_operand(vector)
+        return vector if arel_expression?(vector)
+
+        literal = vector.is_a?(String) ? vector : ParadeDB::Vector.literal(vector)
+        Nodes::TypeCast.new(quoted_value(literal), "vector")
+      end
+
       def proximity_query_node(proximity, boost: nil, const: nil)
         unless proximity.is_a?(ParadeDB::Proximity::Clause)
           raise ArgumentError, "near requires a ParadeDB.proximity(...) clause"

data/lib/parade_db/arel/predications.rb

@@ -120,6 +120,22 @@ module ParadeDB
         ::Arel::Nodes::InfixOperation.new("@@@", self, rhs)
       end
 
+      def pdb_l2_distance(vector)
+        BUILDER.l2_distance(self, vector)
+      end
+
+      def pdb_cosine_distance(vector)
+        BUILDER.cosine_distance(self, vector)
+      end
+
+      def pdb_inner_product(vector)
+        BUILDER.inner_product(self, vector)
+      end
+
+      def pdb_vector_distance(vector, metric: ParadeDB::Vector::DEFAULT_METRIC)
+        BUILDER.vector_distance(self, vector, metric: metric)
+      end
+
       def pdb_score
         ::Arel::Nodes::NamedFunction.new("pdb.score", [self])
       end

data/lib/parade_db/index.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative "tokenizer"
+require_relative "vector"
 
 module ParadeDB
   class Index
@@ -34,7 +35,7 @@ module ParadeDB
       def default_index_name
         return nil if table_name.nil?
 
-        "#{table_name}_bm25_idx"
+        "#{table_name}_search_idx"
       end
 
       def compiled_definition
@@ -82,6 +83,8 @@ module ParadeDB
     # Consumed by migration helpers; validates and normalizes the DSL class
     class DefinitionCompiler
       FIELD_OPTION_KEYS = %i[fast record normalizer expand_dots].freeze
+      INDEX_OPTION_KEYS = %i[target_segment_count centroid_ratio training_samples_per_centroid cluster_replication].freeze
+      POSITIVE_INTEGER_INDEX_OPTION_KEYS = %i[target_segment_count training_samples_per_centroid cluster_replication].freeze
 
       class Compiled
         attr_reader :table_name, :key_field, :index_name, :entries, :index_options, :field_options, :where
@@ -96,7 +99,7 @@ module ParadeDB
           @where = where
         end
       end
-      Entry = Struct.new(:source, :expression, :tokenizer, :options, :query_key, keyword_init: true)
+      Entry = Struct.new(:source, :expression, :tokenizer, :options, :query_key, :metric, keyword_init: true)
 
       class << self
         def compile!(klass)
@@ -153,7 +156,7 @@ module ParadeDB
             end
             normalized = (config || {}).each_with_object({}) { |(k, v), memo| memo[k.to_sym] = v }
 
-            unknown_keys = normalized.keys - (TokenizerParser::TOKENIZER_SINGLE_KEYS + [:tokenizers] + FIELD_OPTION_KEYS)
+            unknown_keys = normalized.keys - (TokenizerParser::TOKENIZER_SINGLE_KEYS + %i[tokenizers metric] + FIELD_OPTION_KEYS)
             unless unknown_keys.empty?
               raise InvalidIndexDefinition,
                     "unknown field config keys for #{source_name.inspect}: #{unknown_keys.map(&:inspect).join(', ')}"
@@ -163,7 +166,27 @@ module ParadeDB
             single_tokenizer_keys_present = TokenizerParser::TOKENIZER_SINGLE_KEYS.any? { |key| normalized.key?(key) }
 
             is_alias = normalized[:alias] && normalized.length == 1
-            if is_alias
+            if normalized.key?(:metric)
+              unless normalized.length == 1
+                raise InvalidIndexDefinition,
+                      "field #{source_name.inspect} cannot mix :metric with other field config keys"
+              end
+
+              begin
+                metric = ParadeDB::Vector.normalize_metric(normalized[:metric])
+              rescue ArgumentError => e
+                raise InvalidIndexDefinition, "field #{source_name.inspect}: #{e.message}"
+              end
+
+              entries << Entry.new(
+                source: source_name,
+                expression: expression?(source_name),
+                tokenizer: nil,
+                options: {},
+                query_key: source_name,
+                metric: metric
+              )
+            elsif is_alias
               entries << Entry.new(
                 source: source_name,
                 expression: expression?(source_name),
@@ -217,16 +240,26 @@ module ParadeDB
             memo[key.to_sym] = value
           end
 
-          unknown = normalized.keys - [:target_segment_count]
+          unknown = normalized.keys - INDEX_OPTION_KEYS
           unless unknown.empty?
             raise InvalidIndexDefinition,
                   "unknown index_options keys: #{unknown.map(&:inspect).join(', ')}"
           end
 
-          if normalized.key?(:target_segment_count)
-            target = normalized[:target_segment_count]
-            unless target.is_a?(Integer) && target.positive?
-              raise InvalidIndexDefinition, "index_options[:target_segment_count] must be an Integer > 0"
+          POSITIVE_INTEGER_INDEX_OPTION_KEYS.each do |key|
+            next unless normalized.key?(key)
+
+            value = normalized[key]
+            unless value.is_a?(Integer) && value.positive?
+              raise InvalidIndexDefinition, "index_options[#{key.inspect}] must be an Integer > 0"
+            end
+          end
+
+          if normalized.key?(:centroid_ratio)
+            ratio = normalized[:centroid_ratio]
+            unless ratio.is_a?(Numeric) && ratio >= 0.000001 && ratio <= 1.0
+              raise InvalidIndexDefinition,
+                    "index_options[:centroid_ratio] must be a Numeric between 0.000001 and 1.0"
             end
           end
 

data/lib/parade_db/migration_helpers.rb

@@ -20,12 +20,12 @@ module ParadeDB
       ensure_postgresql_adapter!
       resolved = resolve_index_klass(index_klass)
       compiled = resolved.compiled_definition
-      remove_bm25_index(compiled.table_name, name: compiled.index_name, if_exists: true)
+      remove_paradedb_index(compiled.table_name, name: compiled.index_name, if_exists: true)
       execute(build_create_sql(compiled, if_not_exists: false))
       remember_schema_index_reference(resolved)
     end
 
-    def add_bm25_index(table, fields:, key_field:, name: nil, index_options: nil, where: nil, if_not_exists: false, concurrently: false)
+    def add_paradedb_index(table, fields:, key_field:, name: nil, index_options: nil, where: nil, if_not_exists: false, concurrently: false)
       ensure_postgresql_adapter!
       anonymous = Class.new(ParadeDB::Index)
       anonymous.table_name = table
@@ -38,31 +38,31 @@ module ParadeDB
       create_paradedb_index(anonymous, if_not_exists: if_not_exists, concurrently: concurrently)
     end
 
-    def remove_bm25_index(table, name: nil, if_exists: false)
+    def remove_paradedb_index(table, name: nil, if_exists: false)
       ensure_postgresql_adapter!
-      index_name = (name || "#{table}_bm25_idx").to_s
+      index_name = (name || "#{table}_search_idx").to_s
       prefix = if_exists ? "IF EXISTS " : ""
       execute("DROP INDEX #{prefix}#{quote_table_name(index_name)}")
     end
 
-    def reindex_bm25(table, name: nil, concurrently: false)
+    def reindex_paradedb_index(table, name: nil, concurrently: false)
       ensure_postgresql_adapter!
       if concurrently && transaction_open_for_paradedb?
-        raise ArgumentError, "reindex_bm25 concurrently: true cannot run inside a transaction"
+        raise ArgumentError, "reindex_paradedb_index concurrently: true cannot run inside a transaction"
       end
 
-      index_name = (name || "#{table}_bm25_idx").to_s
+      index_name = (name || "#{table}_search_idx").to_s
       modifier = concurrently ? " CONCURRENTLY" : ""
       execute("REINDEX INDEX#{modifier} #{quote_table_name(index_name)}")
     end
 
     def dump_paradedb_indexes(stream)
-      rows = paradedb_bm25_index_rows
+      rows = paradedb_index_rows
       return if rows.empty?
 
       stream.puts
       rows.each do |row|
-        ruby_stmt = bm25_index_to_ruby(row)
+        ruby_stmt = paradedb_index_to_ruby(row)
         stream.puts "  #{ruby_stmt}"
       end
     end
@@ -71,8 +71,8 @@ module ParadeDB
       (@paradedb_schema_index_references || []).uniq.sort
     end
 
-    def paradedb_bm25_index_names
-      paradedb_bm25_index_rows.map { |r| r["index_name"] }
+    def paradedb_index_names
+      paradedb_index_rows.map { |r| r["index_name"] }
     end
 
     private
@@ -84,38 +84,41 @@ module ParadeDB
     def build_create_sql(compiled, if_not_exists:, concurrently: false)
       modifier = concurrently ? " CONCURRENTLY" : ""
       prefix = if_not_exists ? "IF NOT EXISTS " : ""
-      fields_sql = compiled.entries.map { |entry| bm25_entry_sql(entry) }.join(", ")
-      with_options_sql = bm25_with_options_sql(compiled)
+      fields_sql = compiled.entries.map { |entry| paradedb_entry_sql(entry) }.join(", ")
+      with_options_sql = paradedb_with_options_sql(compiled)
       where_sql = compiled.where ? "\nWHERE #{compiled.where}" : ""
 
       <<~SQL.strip.gsub(/\s+/, " ")
         CREATE INDEX#{modifier} #{prefix}#{quote_table_name(compiled.index_name)} ON #{quote_table_name(compiled.table_name)}
-        USING bm25 (#{fields_sql})
+        USING paradedb (#{fields_sql})
         WITH (#{with_options_sql})#{where_sql}
       SQL
     end
 
-    def bm25_with_options_sql(compiled)
+    def paradedb_with_options_sql(compiled)
       options = []
       options << "key_field=#{quote(compiled.key_field.to_s)}"
 
       compiled.index_options.each do |key, value|
-        case key.to_sym
-        when :target_segment_count
-          options << "target_segment_count=#{Integer(value)}"
+        name = key.to_sym
+        case name
+        when :target_segment_count, :training_samples_per_centroid, :cluster_replication
+          options << "#{name}=#{Integer(value)}"
+        when :centroid_ratio
+          options << "centroid_ratio=#{Float(value)}"
         else
           raise ParadeDB::InvalidIndexDefinition, "unsupported index option #{key.inspect}"
         end
       end
 
-      bm25_field_option_groups(compiled).each do |param_name, value_hash|
+      paradedb_field_option_groups(compiled).each do |param_name, value_hash|
         options << "#{param_name}=#{quote(JSON.generate(value_hash))}"
       end
 
       options.join(", ")
     end
 
-    def bm25_field_option_groups(compiled)
+    def paradedb_field_option_groups(compiled)
       field_options = compiled.field_options || {}
       return {} if field_options.empty?
 
@@ -128,7 +131,7 @@ module ParadeDB
         column = columns_by_name[source.to_s]
         next unless column
 
-        param_name = bm25_field_option_param_for_column(column)
+        param_name = paradedb_field_option_param_for_column(column)
         next if param_name.nil?
 
         normalized = normalize_field_options_for_param(opts, param_name)
@@ -141,7 +144,7 @@ module ParadeDB
       grouped
     end
 
-    def bm25_field_option_param_for_column(column)
+    def paradedb_field_option_param_for_column(column)
       sql_type = column.sql_type.to_s.downcase
       return "range_fields" if sql_type.include?("range")
 
@@ -182,8 +185,10 @@ module ParadeDB
       end
     end
 
-    def bm25_entry_sql(entry)
-      source_sql = bm25_source_sql(entry)
+    def paradedb_entry_sql(entry)
+      source_sql = paradedb_source_sql(entry)
+
+      return "#{source_sql} #{ParadeDB::Vector::OPCLASSES.fetch(entry.metric)}" if entry.metric
 
       if entry.tokenizer.nil? && entry.query_key != entry.source
         return "(#{source_sql}::pdb.alias(#{quote(entry.query_key)}))"
@@ -194,7 +199,7 @@ module ParadeDB
       "(#{source_sql}::#{tokenizer_sql(entry.tokenizer, entry.options)})"
     end
 
-    def bm25_source_sql(entry)
+    def paradedb_source_sql(entry)
       if entry.expression
         "(#{entry.source})"
       else
@@ -289,12 +294,13 @@ module ParadeDB
       false
     end
 
-    def paradedb_bm25_index_rows
+    def paradedb_index_rows
       sql = <<~SQL
         SELECT
           c.relname  AS index_name,
           t.relname  AS table_name,
           pg_get_indexdef(c.oid) AS indexdef,
+          array_to_json(c.reloptions)::text AS reloptions,
           pg_get_expr(i.indpred, i.indrelid) AS where_clause
         FROM pg_class c
         JOIN pg_namespace n ON n.oid = c.relnamespace
@@ -302,28 +308,28 @@ module ParadeDB
         JOIN pg_class t ON t.oid = i.indrelid
         JOIN pg_am am ON am.oid = c.relam
         WHERE n.nspname = current_schema()
-          AND am.amname = 'bm25'
+          AND am.amname IN ('paradedb', 'bm25')
         ORDER BY t.relname, c.relname
       SQL
       select_all(sql).to_a
     rescue => e
-      Kernel.warn("ParadeDB: unable to query bm25 indexes from catalog: #{e.message}")
+      Kernel.warn("ParadeDB: unable to query paradedb indexes from catalog: #{e.message}")
       []
     end
 
-    def bm25_index_to_ruby(row)
+    def paradedb_index_to_ruby(row)
       indexdef = row["indexdef"]
       table = row["table_name"]
       name = row["index_name"]
 
-      key_field = extract_bm25_key_field(indexdef)
-      index_options = extract_bm25_index_options(indexdef)
-      fields_sql = extract_bm25_fields_sql(indexdef)
-      where = normalize_bm25_where_clause(row["where_clause"])
+      key_field = extract_paradedb_key_field(indexdef)
+      index_options = extract_paradedb_index_options(row["reloptions"])
+      fields_sql = extract_paradedb_fields_sql(indexdef)
+      where = normalize_paradedb_where_clause(row["where_clause"])
 
       if key_field && fields_sql
-        field_sqls = split_bm25_top_level(fields_sql).map(&:strip)
-        parsed = field_sqls.map { |f| bm25_parse_column_entry(f) }
+        field_sqls = split_paradedb_top_level(fields_sql).map(&:strip)
+        parsed = field_sqls.map { |f| paradedb_parse_column_entry(f) }
 
         grouped = {}
         parsed.each { |e| (grouped[e[:source]] ||= []) << e }
@@ -331,17 +337,19 @@ module ParadeDB
         fields_pairs = grouped.map do |source, entries|
           source_ruby = source.match?(/[^a-zA-Z0-9_]/) ? "#{source.inspect} =>" : "#{source}:"
 
-          if entries.all? { |e| e[:tokenizer].nil? }
+          if entries.length == 1 && entries.first[:metric]
+            "#{source_ruby} { metric: #{entries.first[:metric].inspect} }"
+          elsif entries.all? { |e| e[:tokenizer].nil? }
             "#{source_ruby} {}"
           elsif entries.length == 1
-            "#{source_ruby} #{bm25_tokenizer_config_ruby(entries.first)}"
+            "#{source_ruby} #{paradedb_tokenizer_config_ruby(entries.first)}"
           else
-            configs = entries.map { |e| bm25_tokenizer_ruby_from_entry(e) }
+            configs = entries.map { |e| paradedb_tokenizer_ruby_from_entry(e) }
             "#{source_ruby} { tokenizers: [#{configs.join(', ')}] }"
           end
         end
 
-        statement = "add_bm25_index #{table.to_sym.inspect}, " \
+        statement = "add_paradedb_index #{table.to_sym.inspect}, " \
           "fields: { #{fields_pairs.join(', ')} }, " \
           "key_field: #{key_field.to_sym.inspect}, " \
           "name: #{name.inspect}"
@@ -355,7 +363,7 @@ module ParadeDB
       end
     end
 
-    def extract_bm25_key_field(indexdef)
+    def extract_paradedb_key_field(indexdef)
       quoted = indexdef.match(/WITH\s*\([^)]*key_field\s*=\s*'((?:[^']|'')*)'/i)
       return quoted[1].gsub("''", "'") if quoted
 
@@ -365,49 +373,35 @@ module ParadeDB
       nil
     end
 
-    def extract_bm25_index_options(indexdef)
-      with_sql, = extract_bm25_with_components(indexdef)
-      options = {}
-      split_sql_arguments(with_sql).each do |argument|
-        key, value_sql = split_assignment(argument)
-        next if key.nil?
-        next if key == "key_field"
+    def extract_paradedb_index_options(reloptions)
+      paradedb_reloption_entries(reloptions).each_with_object({}) do |entry, options|
+        key, separator, value = entry.to_s.partition("=")
+        next if separator.empty?
 
         case key
-        when "target_segment_count"
-          parsed = parse_sql_literal(value_sql)
-          if parsed.is_a?(Integer)
-            options[:target_segment_count] = parsed
-          elsif parsed.is_a?(String) && parsed.match?(/\A\d+\z/)
-            options[:target_segment_count] = parsed.to_i
-          end
+        when "target_segment_count", "training_samples_per_centroid", "cluster_replication"
+          parsed = Integer(value, 10, exception: false)
+          options[key.to_sym] = parsed if parsed
+        when "centroid_ratio"
+          parsed = Float(value, exception: false)
+          options[:centroid_ratio] = parsed if parsed
         end
       end
-      options
-    rescue
-      {}
     end
 
-    def extract_bm25_fields_sql(indexdef)
-      match = indexdef.match(/USING\s+bm25\s*\(/im)
-
-      start = match.end(0)
-      depth = 1
-      pos = start
-      while pos < indexdef.length && depth > 0
-        case indexdef[pos]
-        when "(" then depth += 1
-        when ")" then depth -= 1
-        end
-        pos += 1
+    def paradedb_reloption_entries(reloptions)
+      case reloptions
+      when Array then reloptions
+      when String then Array(JSON.parse(reloptions))
+      else []
       end
-      raise "Found invalid index definition `#{indexdef}`" if depth != 0
-
-      indexdef[start..pos - 2]
+    rescue JSON::ParserError
+      []
     end
 
-    def extract_bm25_with_components(indexdef)
-      match = indexdef.match(/WITH\s*\(/im)
+    def extract_paradedb_fields_sql(indexdef)
+      match = indexdef.match(/USING\s+(?:paradedb|bm25)\s*\(/im)
+
       start = match.end(0)
       depth = 1
       pos = start
@@ -420,27 +414,23 @@ module ParadeDB
       end
       raise "Found invalid index definition `#{indexdef}`" if depth != 0
 
-      with_sql = indexdef[start..pos - 2]
-      trailing_sql = indexdef[pos..]&.strip
-      trailing_sql = nil if trailing_sql == ""
-
-      [with_sql, trailing_sql]
+      indexdef[start..pos - 2]
     end
 
-    def normalize_bm25_where_clause(where)
+    def normalize_paradedb_where_clause(where)
       return nil if where.nil?
 
       normalized = where.to_s.strip
       return nil if normalized.empty?
 
-      while bm25_wrapped_in_parentheses?(normalized)
+      while paradedb_wrapped_in_parentheses?(normalized)
         normalized = normalized[1...-1].strip
       end
 
       normalized.empty? ? nil : normalized
     end
 
-    def bm25_wrapped_in_parentheses?(sql)
+    def paradedb_wrapped_in_parentheses?(sql)
       return false unless sql.start_with?("(") && sql.end_with?(")")
 
       depth = 0
@@ -456,7 +446,7 @@ module ParadeDB
       depth.zero?
     end
 
-    def split_bm25_top_level(str)
+    def split_paradedb_top_level(str)
       parts = []
       current = +""
       depth = 0
@@ -479,16 +469,28 @@ module ParadeDB
       parts
     end
 
-    def bm25_parse_column_entry(field_sql)
+    def paradedb_parse_column_entry(field_sql)
       stripped = field_sql.strip
 
+      opclass_match = stripped.match(/\A(.+?)\s+(vector_(?:l2|cosine|ip)_ops)\z/m)
+      if opclass_match
+        source_sql = opclass_match[1].strip
+        source_name = identifier_sql?(source_sql) ? unquote_identifier(source_sql) : source_sql
+        return {
+          source: source_name,
+          tokenizer: nil,
+          options: {},
+          metric: ParadeDB::Vector::METRICS_BY_OPCLASS.fetch(opclass_match[2])
+        }
+      end
+
       if stripped.start_with?("(") && stripped.end_with?(")")
         inner = stripped[1..-2].strip
 
         if (source_sql, tok_sql = split_tokenized_cast(inner))
           normalized = unwrap_surrounding_groupings(source_sql)
           source_name = identifier_sql?(normalized) ? unquote_identifier(normalized) : normalized
-          bm25_parse_tokenizer(tok_sql).merge(source: source_name)
+          paradedb_parse_tokenizer(tok_sql).merge(source: source_name)
         else
           { source: inner, tokenizer: nil, options: {} }
         end
@@ -638,7 +640,7 @@ module ParadeDB
       sql.match?(/\A[a-zA-Z_][a-zA-Z0-9_]*\z/) || sql.match?(/\A"(?:[^"]|"")+"\z/)
     end
 
-    def bm25_parse_tokenizer(tokenizer_sql_str)
+    def paradedb_parse_tokenizer(tokenizer_sql_str)
       match = tokenizer_sql_str.strip.match(/\A([a-zA-Z_][a-zA-Z0-9_]*(?:(?:::|\.)[a-zA-Z_][a-zA-Z0-9_]*)*)(?:\((.*)\))?\z/m)
       return { tokenizer: tokenizer_sql_str, options: {} } unless match
 
@@ -677,11 +679,11 @@ module ParadeDB
       { tokenizer: normalized_name, options: options }
     end
 
-    def bm25_tokenizer_config_ruby(entry)
-      "{ tokenizer: #{bm25_tokenizer_ruby_from_entry(entry)} }"
+    def paradedb_tokenizer_config_ruby(entry)
+      "{ tokenizer: #{paradedb_tokenizer_ruby_from_entry(entry)} }"
     end
 
-    def bm25_tokenizer_ruby_from_entry(entry)
+    def paradedb_tokenizer_ruby_from_entry(entry)
       opts = entry[:options].dup
       positional_args = Array(opts.delete(:__positional))
       alias_val = opts.delete(:alias)
@@ -691,10 +693,10 @@ module ParadeDB
 
       opts[:alias] = alias_val if alias_val
 
-      bm25_tokenizer_ruby(entry[:tokenizer], positional_args, opts)
+      paradedb_tokenizer_ruby(entry[:tokenizer], positional_args, opts)
     end
 
-    def bm25_tokenizer_ruby(name, positional_args, options)
+    def paradedb_tokenizer_ruby(name, positional_args, options)
       if name.match?(/\A[a-z_][a-z0-9_]*\z/) && ParadeDB::Tokenizer.respond_to?(name)
         args = positional_args.map { |arg| ruby_literal(arg) }
         args << "options: #{ruby_hash_literal(options)}" unless options.empty?
@@ -866,8 +868,8 @@ if defined?(ActiveRecord::Migration)
         connection.replace_paradedb_index(index_klass)
       end
 
-      def add_bm25_index(table, fields:, key_field:, name: nil, index_options: nil, where: nil, if_not_exists: false, concurrently: false)
-        connection.add_bm25_index(
+      def add_paradedb_index(table, fields:, key_field:, name: nil, index_options: nil, where: nil, if_not_exists: false, concurrently: false)
+        connection.add_paradedb_index(
           table,
           fields: fields,
           key_field: key_field,
@@ -879,12 +881,12 @@ if defined?(ActiveRecord::Migration)
         )
       end
 
-      def remove_bm25_index(table, name: nil, if_exists: false)
-        connection.remove_bm25_index(table, name: name, if_exists: if_exists)
+      def remove_paradedb_index(table, name: nil, if_exists: false)
+        connection.remove_paradedb_index(table, name: name, if_exists: if_exists)
       end
 
-      def reindex_bm25(table, name: nil, concurrently: false)
-        connection.reindex_bm25(table, name: name, concurrently: concurrently)
+      def reindex_paradedb_index(table, name: nil, concurrently: false)
+        connection.reindex_paradedb_index(table, name: name, concurrently: concurrently)
       end
     end
   end
@@ -905,11 +907,11 @@ if defined?(ActiveRecord::SchemaDumper)
       def indexes_in_create(table, stream)
         conn = paradedb_connection
         if conn
-          bm25_names = conn.paradedb_bm25_index_names
+          paradedb_names = conn.paradedb_index_names
           original_indexes = conn.method(:indexes)
 
           conn.define_singleton_method(:indexes) do |tbl|
-            original_indexes.call(tbl).reject { |idx| bm25_names.include?(idx.name) }
+            original_indexes.call(tbl).reject { |idx| paradedb_names.include?(idx.name) }
           end
 
           begin
@@ -940,10 +942,10 @@ if defined?(ActiveRecord::Migration::CommandRecorder)
     module CommandRecorderPatch
       %i[
         create_paradedb_index
-        add_bm25_index
-        remove_bm25_index
+        add_paradedb_index
+        remove_paradedb_index
         replace_paradedb_index
-        reindex_bm25
+        reindex_paradedb_index
       ].each do |method_name|
         define_method(method_name) do |*args, &block|
           record(method_name, args, &block)
@@ -958,10 +960,10 @@ if defined?(ActiveRecord::Migration::CommandRecorder)
         compiled = resolve_paradedb_index_klass(index_klass).compiled_definition
         remove_options = Hash.ruby2_keywords_hash(name: compiled.index_name, if_exists: true)
 
-        [:remove_bm25_index, [compiled.table_name, remove_options]]
+        [:remove_paradedb_index, [compiled.table_name, remove_options]]
       end
 
-      def invert_add_bm25_index(args)
+      def invert_add_paradedb_index(args)
         table, options = args
         options = symbolize_options_hash(options)
 
@@ -969,12 +971,12 @@ if defined?(ActiveRecord::Migration::CommandRecorder)
         remove_options[:name] = options[:name] if options[:name]
         remove_options = Hash.ruby2_keywords_hash(remove_options)
 
-        [:remove_bm25_index, [table, remove_options]]
+        [:remove_paradedb_index, [table, remove_options]]
       end
 
-      def invert_remove_bm25_index(_args)
+      def invert_remove_paradedb_index(_args)
         raise ActiveRecord::IrreversibleMigration,
-              "remove_bm25_index is not automatically reversible. Use #up/#down or #reversible."
+              "remove_paradedb_index is not automatically reversible. Use #up/#down or #reversible."
       end
 
       def invert_replace_paradedb_index(_args)
@@ -982,9 +984,9 @@ if defined?(ActiveRecord::Migration::CommandRecorder)
               "replace_paradedb_index is not automatically reversible. Use #up/#down or #reversible."
       end
 
-      def invert_reindex_bm25(_args)
+      def invert_reindex_paradedb_index(_args)
         raise ActiveRecord::IrreversibleMigration,
-              "reindex_bm25 is not automatically reversible. Use #up/#down or #reversible."
+              "reindex_paradedb_index is not automatically reversible. Use #up/#down or #reversible."
       end
 
       def resolve_paradedb_index_klass(index_klass)

data/lib/parade_db/model.rb

@@ -10,6 +10,7 @@ module ParadeDB
     INJECTED_CLASS_METHODS = [
       :paradedb_search,
       :more_like_this,
+      :nearest,
       :with_facets,
       :facets,
       :with_agg,
@@ -67,6 +68,12 @@ module ParadeDB
         all.extending(SearchMethods).more_like_this(key, fields: fields, **options)
       end
 
+      def nearest(column, vector, metric: nil)
+        ensure_postgres!
+        paradedb_validate_index!
+        all.extending(SearchMethods).nearest(column, vector, metric: metric)
+      end
+
       def with_facets(*fields, **opts)
         ensure_postgres!
         paradedb_validate_index!
@@ -158,7 +165,7 @@ module ParadeDB
           next if paradedb_catalog_index_valid?(definition)
 
           all_valid = false
-          message = "ParadeDB index drift detected for #{name}: expected #{definition.index_name} on #{definition.table_name} with bm25."
+          message = "ParadeDB index drift detected for #{name}: expected #{definition.index_name} on #{definition.table_name} with a ParadeDB access method."
           case ParadeDB.index_validation_mode
           when :warn
             paradedb_log_warn(message)
@@ -263,7 +270,7 @@ module ParadeDB
           WHERE c.relname = #{connection.quote(definition.index_name.to_s)}
             AND t.relname = #{connection.quote(definition.table_name.to_s)}
             AND n.nspname = current_schema()
-            AND am.amname = 'bm25'
+            AND am.amname IN ('paradedb', 'bm25')
           LIMIT 1
         SQL
 

data/lib/parade_db/search_methods.rb

@@ -479,6 +479,19 @@ module ParadeDB
       rel.except(:select, :group).select(*group_nodes, *aggregate_nodes).group(*group_nodes)
     end
 
+    # Orders by vector distance for Top-K pushdown inside the ParadeDB index.
+    # Adds `key_field @@@ pdb.all()` when the relation has no ParadeDB predicate,
+    # since vector ordering requires a @@@ predicate to activate the index scan.
+    # Callers must add `.limit(k)`; the metric defaults to the index opclass metric.
+    def nearest(column, vector, metric: nil)
+      ensure_paradedb_runtime!
+      resolved_metric = metric || index_vector_metric(column) || ParadeDB::Vector::DEFAULT_METRIC
+      node = builder.vector_distance(column, vector, metric: resolved_metric)
+
+      rel = has_paradedb_predicate? ? self : ensure_paradedb_predicate
+      rel.order(node.asc)
+    end
+
     def has_paradedb_predicate?
       PredicateInspector.relation_has_paradedb_predicate?(self)
     end
@@ -490,6 +503,12 @@ module ParadeDB
 
     private
 
+    def index_vector_metric(column)
+      return nil unless klass.respond_to?(:paradedb_index_entry, true)
+
+      klass.send(:paradedb_index_entry, column)&.metric
+    end
+
     def paradedb_runtime_key_field
       return primary_key unless klass.respond_to?(:paradedb_key_field)
 

data/lib/parade_db/vector.rb

@@ -0,0 +1,119 @@
+# frozen_string_literal: true
+
+require "active_model"
+
+module ParadeDB
+  class Vector < ActiveModel::Type::Value
+    METRICS = %i[l2 cosine ip].freeze
+    METRIC_ALIASES = { inner_product: :ip }.freeze
+    OPCLASSES = { l2: "vector_l2_ops", cosine: "vector_cosine_ops", ip: "vector_ip_ops" }.freeze
+    METRICS_BY_OPCLASS = OPCLASSES.invert.freeze
+    DISTANCE_OPERATORS = { l2: "<->", cosine: "<=>", ip: "<#>" }.freeze
+    DEFAULT_METRIC = :l2
+
+    def self.normalize_metric(metric)
+      normalized = metric.to_sym
+      normalized = METRIC_ALIASES.fetch(normalized, normalized)
+      return normalized if METRICS.include?(normalized)
+
+      raise ArgumentError,
+            "unknown vector metric #{metric.inspect}. Valid metrics: #{(METRICS + METRIC_ALIASES.keys).map(&:inspect).join(', ')}"
+    end
+
+    def self.literal(value)
+      "[#{Array(value).map { |v| Float(v) }.join(',')}]"
+    end
+
+    attr_reader :limit
+
+    def initialize(limit: nil)
+      super()
+      @limit = limit
+    end
+
+    def type
+      :vector
+    end
+
+    def cast(value)
+      case value
+      when nil
+        nil
+      when Array
+        validate_dimensions!(value.map { |v| Float(v) })
+      when String
+        cast_string(value)
+      else
+        raise ArgumentError, "cannot cast #{value.class} to vector"
+      end
+    end
+
+    def serialize(value)
+      return nil if value.nil?
+
+      self.class.literal(cast(value))
+    end
+
+    def deserialize(value)
+      cast(value)
+    end
+
+    def changed_in_place?(raw_old_value, new_value)
+      deserialize(raw_old_value) != new_value
+    end
+
+    def self.register_with_type_map(m)
+      m.register_type "vector" do |_, _, sql_type|
+        limit = sql_type.to_s[/\((\d+)\)/, 1]
+        ParadeDB::Vector.new(limit: limit&.to_i)
+      end
+    end
+
+    module PostgreSQLAdapterPatch
+      def initialize_type_map(m)
+        super
+        ParadeDB::Vector.register_with_type_map(m)
+      end
+    end
+
+    def self.install_postgresql_adapter!(adapter_class)
+      adapter_class::NATIVE_DATABASE_TYPES[:vector] = { name: "vector" }
+
+      unless adapter_class.singleton_class.ancestors.include?(PostgreSQLAdapterPatch)
+        adapter_class.singleton_class.prepend(PostgreSQLAdapterPatch)
+      end
+
+      register_with_type_map(adapter_class::TYPE_MAP) if adapter_class.const_defined?(:TYPE_MAP)
+
+      table_definition = ActiveRecord::ConnectionAdapters::PostgreSQL::TableDefinition
+      unless table_definition.method_defined?(:vector)
+        table_definition.send(:define_column_methods, :vector)
+      end
+    end
+
+    private
+
+    def cast_string(value)
+      stripped = value.strip
+      unless stripped.start_with?("[") && stripped.end_with?("]")
+        raise ArgumentError, "malformed vector literal: #{value.inspect}"
+      end
+
+      inner = stripped[1..-2].strip
+      values = inner.empty? ? [] : inner.split(",").map { |v| Float(v) }
+      validate_dimensions!(values)
+    end
+
+    def validate_dimensions!(values)
+      if limit && values.length != limit
+        raise ArgumentError, "expected #{limit} dimensions, got #{values.length}"
+      end
+
+      values
+    end
+  end
+end
+
+ActiveSupport.on_load(:active_record_postgresqladapter) do
+  ParadeDB::Vector.install_postgresql_adapter!(self)
+end

data/lib/parade_db/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ParadeDB
-  VERSION = "0.9.0"
+  VERSION = "0.11.0"
 end

data/lib/parade_db.rb

@@ -3,6 +3,7 @@
 require_relative "parade_db/version"
 require_relative "parade_db/errors"
 require_relative "parade_db/arel"
+require_relative "parade_db/vector"
 require_relative "parade_db/index"
 require_relative "parade_db/aggregations"
 require_relative "parade_db/proximity"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails-paradedb
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 0.11.0
 platform: ruby
 authors:
 - ParadeDB
@@ -115,6 +115,7 @@ files:
 - lib/parade_db/search_methods.rb
 - lib/parade_db/tokenizer.rb
 - lib/parade_db/tokenizer_sql.rb
+- lib/parade_db/vector.rb
 - lib/parade_db/version.rb
 homepage: https://github.com/paradedb/rails-paradedb
 licenses: