searchkick 5.4.0 → 5.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5dfa66d383b0e1a91288b4636fef42b22ac73ec63294a7955ac20298851df1c1
-  data.tar.gz: fbe59c3e85352b01674c16831f67b3906367e6a032769582ee552038853194f8
+  metadata.gz: 37b82bea130dfffde9634bf74e4e078e97bbe6e8d0b4e540c13800c765ddc6c3
+  data.tar.gz: 2435f38ff18bf471673e1df11f0fd96cfacb408cf390e04afcc068488777a358
 SHA512:
-  metadata.gz: 25d7df6cf3861522a99851c8c561f0726a6e922d4aafb0fe646883b86c37ba4c1ec78735940a5eb70056433422fa2987989701260ec7b1e2969fb452758d2d43
-  data.tar.gz: d9b637bba6c90e1c08de0b9cd8239e6b8b03aa86eb0a77aebc6b855ff0744220d775c8b7c8a12e1df736f98ee44dac3c6f11ad3a0506aa51e5c22f120f3c13d3
+  metadata.gz: f0541cc6673c2382ec82bce16bb5c230fa0fb3220d6430324739c8fb722ca75f8f70d33c1833642ee03edc936f8dba88c37282cdf6e200ead84ae16688054307
+  data.tar.gz: 7522e671cf005d61a5df2856e6754605570f281f6e4b359f537f4a9d6cfe17eccad1466b81adbb683b782c9941caac4f935a2244094f7af11b09f58954d173ff

data/CHANGELOG.md

@@ -1,3 +1,11 @@
+## 5.5.0 (2025-04-03)
+
+- Added `m` and `ef_construction` to `knn` index option
+- Added `ef_search` to `knn` search option
+- Fixed exact cosine distance for OpenSearch 2.19+
+- Dropped support for Ruby < 3.2 and Active Record < 7.1
+- Dropped support for Mongoid < 8
+
 ## 5.4.0 (2024-09-04)
 
 - Added `knn` option
@@ -833,4 +841,22 @@ Breaking changes
 
 ## 0.1.2 (2013-07-30)
 
-- Launch
+- Use conversions by default
+
+## 0.1.1 (2013-07-29)
+
+- Renamed `_source` to `search_data`
+- Renamed `searchkick_import` to `search_import`
+
+## 0.1.0 (2013-07-28)
+
+- Added `_source` method
+- Added `index_name` option
+
+## 0.0.2 (2013-07-17)
+
+- Added `conversions` option
+
+## 0.0.1 (2013-07-14)
+
+- First release

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013-2023 Andrew Kane
+Copyright (c) 2013-2025 Andrew Kane
 
 MIT License
 

data/README.md

@@ -821,7 +821,7 @@ Autocomplete predicts what a user will type, making the search experience faster
 
 **Note 2:** If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s *much* faster to load all records into JavaScript and autocomplete there (eliminates network requests).
 
-First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.
+First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing fast queries.
 
 ```ruby
 class Movie < ApplicationRecord
@@ -835,7 +835,7 @@ Reindex and search with:
 Movie.search("jurassic pa", fields: [:title], match: :word_start)
 ```
 
-Typically, you want to use a JavaScript library like [typeahead.js](https://twitter.github.io/typeahead.js/) or [jQuery UI](https://jqueryui.com/autocomplete/).
+Use a front-end library like [typeahead.js](https://twitter.github.io/typeahead.js/) to show the results.
 
 #### Here’s how to make it work with Rails
 
@@ -844,13 +844,14 @@ First, add a route and controller action.
 ```ruby
 class MoviesController < ApplicationController
   def autocomplete
-    render json: Movie.search(params[:query], {
+    render json: Movie.search(
+      params[:query],
       fields: ["title^5", "director"],
       match: :word_start,
       limit: 10,
       load: false,
       misspellings: {below: 5}
-    }).map(&:title)
+    ).map(&:title)
   end
 end
 ```
@@ -1023,7 +1024,7 @@ Additional options can be specified for each field:
 Band.search("cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}})
 ```
 
-You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html).
+You can find available highlight options in the [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html) or [OpenSearch](https://opensearch.org/docs/latest/search-plugins/searching-data/highlight/) reference.
 
 ## Similar Items
 
@@ -1511,13 +1512,25 @@ See [Production Rails](https://github.com/ankane/production_rails) for other goo
 
 ### JSON Generation
 
-Significantly increase performance with faster JSON generation. Add [Oj](https://github.com/ohler55/oj) to your Gemfile.
+Increase performance with faster JSON generation. Add to your Gemfile:
 
 ```ruby
-gem "oj"
+gem "json", ">= 2.10.2"
 ```
 
-This speeds up all JSON generation and parsing in your application (automatically!)
+And create an initializer with:
+
+```ruby
+class SearchSerializer
+  def dump(object)
+    JSON.generate(object)
+  end
+end
+
+Elasticsearch::API.settings[:serializer] = SearchSerializer.new
+# or
+OpenSearch::API.settings[:serializer] = SearchSerializer.new
+```
 
 ### Persistent HTTP Connections
 
@@ -1533,8 +1546,6 @@ To reduce log noise, create an initializer with:
 Ethon.logger = Logger.new(nil)
 ```
 
-If you run into issues on Windows, check out [this post](https://www.rastating.com/fixing-issues-in-typhoeus-and-httparty-on-windows/).
-
 ### Searchable Fields
 
 By default, all string fields are searchable (can be used in `fields` option). Speed up indexing and reduce index size by only making some fields searchable.
@@ -1563,7 +1574,7 @@ For large data sets, you can use background jobs to parallelize reindexing.
 
 ```ruby
 Product.reindex(mode: :async)
-# {index_name: "products_production_20170111210018065"}
+# {index_name: "products_production_20250111210018065"}
 ```
 
 Once the jobs complete, promote the new index with:
@@ -1590,19 +1601,19 @@ You can also have Searchkick wait for reindexing to complete
 Product.reindex(mode: :async, wait: true)
 ```
 
-You can use [ActiveJob::TrafficControl](https://github.com/nickelser/activejob-traffic_control) to control concurrency. Install the gem:
+You can use your background job framework to control concurrency. For Solid Queue, create an initializer with:
 
 ```ruby
-gem "activejob-traffic_control", ">= 0.1.3"
-```
-
-And create an initializer with:
+module SearchkickBulkReindexConcurrency
+  extend ActiveSupport::Concern
 
-```ruby
-ActiveJob::TrafficControl.client = Searchkick.redis
+  included do
+    limits_concurrency to: 3, key: ""
+  end
+end
 
-class Searchkick::BulkReindexJob
-  concurrency 3
+Rails.application.config.after_initialize do
+  Searchkick::BulkReindexJob.include(SearchkickBulkReindexConcurrency)
 end
 ```
 
@@ -1616,7 +1627,7 @@ You can specify a longer refresh interval while reindexing to increase performan
 Product.reindex(mode: :async, refresh_interval: "30s")
 ```
 
-**Note:** This only makes a noticable difference with parallel reindexing.
+**Note:** This only makes a noticeable difference with parallel reindexing.
 
 When promoting, have it restored to the value in your mapping (defaults to `1s`).
 
@@ -1863,9 +1874,27 @@ Reindex and search with:
 Product.search(knn: {field: :embedding, vector: [1, 2, 3]}, limit: 10)
 ```
 
+### HNSW Options
+
+Nearest neighbor search uses [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) for indexing.
+
+Specify `m` and `ef_construction`
+
+```ruby
+class Product < ApplicationRecord
+  searchkick knn: {embedding: {dimensions: 3, distance: "cosine", m: 16, ef_construction: 100}}
+end
+```
+
+Specify `ef_search`
+
+```ruby
+Product.search(knn: {field: :embedding, vector: [1, 2, 3], ef_search: 40}, limit: 10)
+```
+
 ## Semantic Search
 
-First, add [nearest neighbor search](#nearest-neighbor-search-unreleased-experimental) to your model
+First, add [nearest neighbor search](#nearest-neighbor-search) to your model
 
 ```ruby
 class Product < ApplicationRecord
@@ -2205,42 +2234,6 @@ end
 
 For convenience, this is set by default in the test environment.
 
-## Upgrading
-
-### 5.0
-
-Searchkick 5 supports both the `elasticsearch` and `opensearch-ruby` gems. Add the one you want to use to your Gemfile:
-
-```ruby
-gem "elasticsearch"
-# or
-gem "opensearch-ruby"
-```
-
-If using the deprecated `faraday_middleware-aws-signers-v4` gem, switch to `faraday_middleware-aws-sigv4`.
-
-Also, searches now use lazy loading:
-
-```ruby
-# search not executed
-Product.search("milk")
-
-# search executed
-Product.search("milk").to_a
-```
-
-You can reindex relations in the background:
-
-```ruby
-store.products.reindex(mode: :async)
-# or
-store.products.reindex(mode: :queue)
-```
-
-And there’s a [new option](#default-scopes) for models with default scopes.
-
-Check out the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md#500-2022-02-21) for the full list of changes.
-
 ## History
 
 View the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md).

data/lib/searchkick/index_options.rb

@@ -451,7 +451,8 @@ module Searchkick
             vector_options[:method] = {
               name: "hnsw",
               space_type: space_type,
-              engine: "lucene"
+              engine: "lucene",
+              parameters: knn_options.slice(:m, :ef_construction)
             }
           end
 
@@ -475,6 +476,14 @@ module Searchkick
               else
                 raise ArgumentError, "Unknown distance: #{distance}"
               end
+
+            vector_index_options = knn_options.slice(:m, :ef_construction)
+            if vector_index_options.any?
+              # TODO no quantization by default in Searchkick 6
+              # int8_hnsw was made the default in Elasticsearch 8.14.0
+              type = Searchkick.server_below?("8.14.0") ? "hnsw" : "int8_hnsw"
+              vector_options[:index_options] = {type: type}.merge(vector_index_options)
+            end
           end
 
           mapping[field.to_s] = vector_options

data/lib/searchkick/query.rb

@@ -891,6 +891,7 @@ module Searchkick
       exact = knn[:exact]
       exact = field_options[:distance].nil? || distance != field_options[:distance] if exact.nil?
       k = per_page + offset
+      ef_search = knn[:ef_search]
       filter = payload.delete(:query)
 
       if distance.nil?
@@ -934,22 +935,31 @@ module Searchkick
                   space_type: space_type
                 }
               },
-              boost: distance == "cosine" ? 0.5 : 1.0
+              boost: distance == "cosine" && Searchkick.server_below?("2.19.0", true) ? 0.5 : 1.0
             }
           }
         else
+          if ef_search && Searchkick.server_below?("2.16.0", true)
+            raise Error, "ef_search requires OpenSearch 2.16+"
+          end
+
           payload[:query] = {
             knn: {
               field.to_sym => {
                 vector: vector,
                 k: k,
                 filter: filter
-              }
+              }.merge(ef_search ? {method_parameters: {ef_search: ef_search}} : {})
             }
           }
         end
       else
         if exact
+          # prevent incorrect distances/results with Elasticsearch 9.0.0-rc1
+          if !below90? && field_options[:distance] == "cosine" && distance != "cosine"
+            raise ArgumentError, "distance must match searchkick options"
+          end
+
           # https://github.com/elastic/elasticsearch/blob/main/docs/reference/vectors/vector-functions.asciidoc
           source =
             case distance
@@ -987,7 +997,7 @@ module Searchkick
             query_vector: vector,
             k: k,
             filter: filter
-          }
+          }.merge(ef_search ? {num_candidates: ef_search} : {})
         end
       end
     end
@@ -1134,13 +1144,17 @@ module Searchkick
                 range_query =
                   case op
                   when :gt
-                    {from: op_value, include_lower: false}
+                    # TODO always use gt in Searchkick 6
+                    below90? ? {from: op_value, include_lower: false} : {gt: op_value}
                   when :gte
-                    {from: op_value, include_lower: true}
+                    # TODO always use gte in Searchkick 6
+                    below90? ? {from: op_value, include_lower: true} : {gte: op_value}
                   when :lt
-                    {to: op_value, include_upper: false}
+                    # TODO always use lt in Searchkick 6
+                    below90? ? {to: op_value, include_upper: false} : {lt: op_value}
                   when :lte
-                    {to: op_value, include_upper: true}
+                    # TODO always use lte in Searchkick 6
+                    below90? ? {to: op_value, include_upper: true} : {lte: op_value}
                   else
                     raise ArgumentError, "Unknown where operator: #{op.inspect}"
                   end
@@ -1301,5 +1315,9 @@ module Searchkick
     def below80?
       Searchkick.server_below?("8.0.0")
     end
+
+    def below90?
+      Searchkick.server_below?("9.0.0")
+    end
   end
 end

data/lib/searchkick/railtie.rb

@@ -1,4 +1,4 @@
-module Searckick
+module Searchkick
   class Railtie < Rails::Railtie
     rake_tasks do
       load "tasks/searchkick.rake"

data/lib/searchkick/version.rb

@@ -1,3 +1,3 @@
 module Searchkick
-  VERSION = "5.4.0"
+  VERSION = "5.5.0"
 end

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: searchkick
 version: !ruby/object:Gem::Version
-  version: 5.4.0
+  version: 5.5.0
 platform: ruby
 authors:
 - Andrew Kane
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-09-04 00:00:00.000000000 Z
+date: 2025-04-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activemodel
@@ -16,14 +15,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '6.1'
+        version: '7.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '6.1'
+        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: hashie
   requirement: !ruby/object:Gem::Requirement
@@ -38,7 +37,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description:
 email: andrew@ankane.org
 executables: []
 extensions: []
@@ -79,7 +77,6 @@ homepage: https://github.com/ankane/searchkick
 licenses:
 - MIT
 metadata: {}
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -87,15 +84,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.1'
+      version: '3.2'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.11
-signing_key:
+rubygems_version: 3.6.2
 specification_version: 4
 summary: Intelligent search made easy with Rails and Elasticsearch or OpenSearch
 test_files: []