searchkick 5.0.2 → 5.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5ebb326348913a8532f1f4e7771bcdb57dda1abe4cc668e4f5bf8fe44bdbc85b
-  data.tar.gz: 57b63c4444b9dbe26bb8decc4ab4dca81e4e037e6642d5728b96549aea0b2131
+  metadata.gz: 5dfa66d383b0e1a91288b4636fef42b22ac73ec63294a7955ac20298851df1c1
+  data.tar.gz: fbe59c3e85352b01674c16831f67b3906367e6a032769582ee552038853194f8
 SHA512:
-  metadata.gz: f901dfae13328bb168e60cc72559ee2fd624f801456fee1628272c83ed402c4d00a83d3cac2ba978e6ce67c76853ca2dc5c1c0048b11ac34d74bb2b18bc3b974
-  data.tar.gz: '086eb84edef491f27c254cb09ee5bce9ec07bcec46f79b2ca65404041e67bfccd877e30e92e6393ca2117c353ec71e8c632c50615c71e481e89c30c071934dd5'
+  metadata.gz: 25d7df6cf3861522a99851c8c561f0726a6e922d4aafb0fe646883b86c37ba4c1ec78735940a5eb70056433422fa2987989701260ec7b1e2969fb452758d2d43
+  data.tar.gz: d9b637bba6c90e1c08de0b9cd8239e6b8b03aa86eb0a77aebc6b855ff0744220d775c8b7c8a12e1df736f98ee44dac3c6f11ad3a0506aa51e5c22f120f3c13d3

data/CHANGELOG.md

@@ -1,3 +1,75 @@
+## 5.4.0 (2024-09-04)
+
+- Added `knn` option
+- Added `rrf` method
+- Added experimental support for scripting to `where` option
+- Added warning for `exists` with non-`true` values
+- Added warning for full reindex and `:queue` mode
+- Fixed `per_page` method when paginating beyond `max_result_window`
+- Dropped support for Ruby < 3.1
+
+## 5.3.1 (2023-11-28)
+
+- Fixed error with misspellings below and failed queries
+
+## 5.3.0 (2023-07-02)
+
+- Fixed error with `cutoff_frequency`
+- Dropped support for Ruby < 3 and Active Record < 6.1
+- Dropped support for Mongoid < 7
+
+## 5.2.4 (2023-05-11)
+
+- Fixed error with non-string routing and `:async` mode
+
+## 5.2.3 (2023-04-12)
+
+- Fixed error with missing records and multiple models
+
+## 5.2.2 (2023-04-01)
+
+- Fixed `total_docs` method
+- Fixed deprecation warning with Active Support 7.1
+
+## 5.2.1 (2023-02-21)
+
+- Added support for `redis-client` gem
+
+## 5.2.0 (2023-02-08)
+
+- Added model name to warning about missing records
+- Fixed unnecessary data loading when reindexing relations with `:async` and `:queue` modes
+
+## 5.1.2 (2023-01-29)
+
+- Fixed error with missing point in time
+
+## 5.1.1 (2022-12-05)
+
+- Added support for strings for `offset` and `per_page`
+
+## 5.1.0 (2022-10-12)
+
+- Added support for fractional search timeout
+- Fixed search timeout with `elasticsearch` 8+ and `opensearch-ruby` gems
+- Fixed search timeout not applying to `multi_search`
+
+## 5.0.5 (2022-10-09)
+
+- Added `model` method to `Searchkick::Relation`
+- Fixed deprecation warning with `redis` gem
+- Fixed `respond_to?` method on relation loading relation
+- Fixed `Relation loaded` error for non-mutating methods on relation
+
+## 5.0.4 (2022-06-16)
+
+- Added `max_result_window` option
+- Improved error message for unsupported versions of Elasticsearch
+
+## 5.0.3 (2022-03-13)
+
+- Fixed context for index name for inherited models
+
 ## 5.0.2 (2022-03-03)
 
 - Fixed index name for inherited models

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013-2021 Andrew Kane
+Copyright (c) 2013-2023 Andrew Kane
 
 MIT License
 

data/README.md

@@ -26,7 +26,7 @@ Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Au
 
 :tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
 
-[![Build Status](https://github.com/ankane/searchkick/workflows/build/badge.svg?branch=master)](https://github.com/ankane/searchkick/actions)
+[![Build Status](https://github.com/ankane/searchkick/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/searchkick/actions)
 
 ## Contents
 
@@ -43,15 +43,13 @@ Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Au
 - [Reference](#reference)
 - [Contributing](#contributing)
 
-Searchkick 5.0 was recently released! See [how to upgrade](#upgrading)
-
 ## Getting Started
 
 Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html). For Homebrew, use:
 
 ```sh
-brew install elasticsearch
-brew services start elasticsearch
+brew install elastic/tap/elasticsearch-full
+brew services start elasticsearch-full
 # or
 brew install opensearch
 brew services start opensearch
@@ -66,7 +64,7 @@ gem "elasticsearch"   # select one
 gem "opensearch-ruby" # select one
 ```
 
-The latest version works with Elasticsearch 7 and 8 and OpenSearch 1. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
+The latest version works with Elasticsearch 7 and 8 and OpenSearch 1 and 2. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
 
 Add searchkick to models you want to search.
 
@@ -122,9 +120,9 @@ where: {
   category: /frozen .+/,         # regexp
   category: {prefix: "frozen"},  # prefix
   store_id: {exists: true},      # exists
+  _not: {store_id: 1},           # negate a condition
   _or: [{in_stock: true}, {backordered: true}],
-  _and: [{in_stock: true}, {backordered: true}],
-  _not: {store_id: 1}            # negate a condition
+  _and: [{in_stock: true}, {backordered: true}]
 }
 ```
 
@@ -292,12 +290,18 @@ Option | Matches | Example
 
 The default is `:word`. The most matches will happen with `:word_middle`.
 
+To specify different matching for different fields, use:
+
+```ruby
+Product.search(query, fields: [{name: :word_start}, {brand: :word_middle}])
+```
+
 ### Exact Matches
 
 To match a field exactly (case-sensitive), use:
 
 ```ruby
-Product.search(query, fields: [{email: :exact}, :name])
+Product.search(query, fields: [{name: :exact}])
 ```
 
 ### Phrase Matches
@@ -323,11 +327,11 @@ end
 See the [list of languages](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html#analysis-stemmer-tokenfilter-configure-parms). A few languages require plugins:
 
 - `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)
-- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-smartcn.html)
-- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-kuromoji.html)
+- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html)
+- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html)
 - `korean` - [analysis-openkoreantext plugin](https://github.com/open-korean-text/elasticsearch-analysis-openkoreantext)
-- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-nori.html)
-- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-stempel.html)
+- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-nori.html)
+- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-stempel.html)
 - `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html)
 - `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)
 
@@ -592,6 +596,14 @@ There are four strategies for keeping the index synced with your database.
   end
   ```
 
+  And reindex a record or relation manually.
+
+  ```ruby
+  product.reindex
+  # or
+  store.products.reindex(mode: :async)
+  ```
+
 You can also do bulk updates.
 
 ```ruby
@@ -608,6 +620,12 @@ Searchkick.callbacks(false) do
 end
 ```
 
+Or override the model’s strategy.
+
+```ruby
+product.reindex(mode: :async) # :inline or :queue
+```
+
 ### Associations
 
 Data is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:
@@ -654,20 +672,16 @@ The best starting point to improve your search **by far** is to track searches a
 Product.search("apple", track: {user_id: current_user.id})
 ```
 
-[See the docs](https://github.com/ankane/searchjoy) for how to install and use.
-
-Focus on:
+[See the docs](https://github.com/ankane/searchjoy) for how to install and use. Focus on top searches with a low conversion rate.
 
-- top searches with low conversions
-- top searches with no results
-
-Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
+Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches. This can make a huge difference on the quality of your search.
 
 Add conversion data with:
 
 ```ruby
 class Product < ApplicationRecord
-  has_many :searches, class_name: "Searchjoy::Search", as: :convertable
+  has_many :conversions, class_name: "Searchjoy::Conversion", as: :convertable
+  has_many :searches, class_name: "Searchjoy::Search", through: :conversions
 
   searchkick conversions: [:conversions] # name of field
 
@@ -681,15 +695,100 @@ class Product < ApplicationRecord
 end
 ```
 
-Reindex and set up a cron job to add new conversions daily.
+Reindex and set up a cron job to add new conversions daily. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
+
+### Performant Conversions
+
+A performant way to do conversions is to cache them to prevent N+1 queries. For Postgres, create a migration with:
 
 ```ruby
-rake searchkick:reindex CLASS=Product
+add_column :products, :search_conversions, :jsonb
 ```
 
-This can make a huge difference on the quality of your search.
+For MySQL, use `:json`, and for others, use `:text` with a [JSON serializer](https://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Serialization/ClassMethods.html).
+
+Next, update your model. Create a separate method for conversion data so you can use [partial reindexing](#partial-reindexing).
 
-For a more performant way to reindex conversion data, check out [performant conversions](#performant-conversions).
+```ruby
+class Product < ApplicationRecord
+  searchkick conversions: [:conversions]
+
+  def search_data
+    {
+      name: name,
+      category: category
+    }.merge(conversions_data)
+  end
+
+  def conversions_data
+    {
+      conversions: search_conversions || {}
+    }
+  end
+end
+```
+
+Deploy and reindex your data. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
+
+```ruby
+Product.reindex
+```
+
+Then, create a job to update the conversions column and reindex records with new conversions. Here’s one you can use for Searchjoy:
+
+```ruby
+class UpdateConversionsJob < ApplicationJob
+  def perform(class_name, since: nil, update: true, reindex: true)
+    model = Searchkick.load_model(class_name)
+
+    # get records that have a recent conversion
+    recently_converted_ids =
+      Searchjoy::Conversion.where(convertable_type: class_name).where(created_at: since..)
+      .order(:convertable_id).distinct.pluck(:convertable_id)
+
+    # split into batches
+    recently_converted_ids.in_groups_of(1000, false) do |ids|
+      if update
+        # fetch conversions
+        conversions =
+          Searchjoy::Conversion.where(convertable_id: ids, convertable_type: class_name)
+          .joins(:search).where.not(searchjoy_searches: {user_id: nil})
+          .group(:convertable_id, :query).distinct.count(:user_id)
+
+        # group by record
+        conversions_by_record = {}
+        conversions.each do |(id, query), count|
+          (conversions_by_record[id] ||= {})[query] = count
+        end
+
+        # update conversions column
+        model.transaction do
+          conversions_by_record.each do |id, conversions|
+            model.where(id: id).update_all(search_conversions: conversions)
+          end
+        end
+      end
+
+      if reindex
+        # reindex conversions data
+        model.where(id: ids).reindex(:conversions_data)
+      end
+    end
+  end
+end
+```
+
+Run the job:
+
+```ruby
+UpdateConversionsJob.perform_now("Product")
+```
+
+And set it up to run daily.
+
+```ruby
+UpdateConversionsJob.perform_later("Product", since: 1.day.ago)
+```
 
 ## Personalized Results
 
@@ -716,7 +815,7 @@ Product.search("milk", boost_where: {orderer_ids: current_user.id})
 
 Autocomplete predicts what a user will type, making the search experience faster and easier.
 
-![Autocomplete](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/autocomplete.png)
+![Autocomplete](https://gist.githubusercontent.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/autocomplete.png)
 
 **Note:** To autocomplete on search terms rather than results, check out [Autosuggest](https://github.com/ankane/autosuggest).
 
@@ -782,7 +881,7 @@ Then add the search box and JavaScript code to a view.
 
 ## Suggestions
 
-![Suggest](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/recursion.png)
+![Suggest](https://gist.githubusercontent.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/recursion.png)
 
 ```ruby
 class Product < ApplicationRecord
@@ -801,7 +900,7 @@ products.suggestions # ["peanut butter"]
 
 [Aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) provide aggregated search data.
 
-![Aggregations](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/facets.png)
+![Aggregations](https://gist.githubusercontent.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/facets.png)
 
 ```ruby
 products = Product.search("chuck taylor", aggs: [:product_type, :gender, :brand])
@@ -1011,7 +1110,7 @@ Restaurant.search("soup", where: {bounds: {geo_shape: {type: "polygon", coordina
 Falling entirely within the query shape
 
 ```ruby
-Restaurant.search("salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: [{lat: 38, lon: -123}], radius: "1km"}}})
+Restaurant.search("salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: {lat: 38, lon: -123}, radius: "1km"}}})
 ```
 
 Not touching the query shape
@@ -1384,7 +1483,15 @@ ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@ho
 ENV["OPENSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
 ```
 
-See [elastic-transport](https://github.com/elastic/elastic-transport-ruby) or [opensearch-transport](https://github.com/opensearch-project/opensearch-ruby/tree/main/opensearch-transport) for a complete list of options.
+### Client Options
+
+Create an initializer with:
+
+```ruby
+Searchkick.client_options[:reload_connections] = true
+```
+
+See the docs for [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/advanced-config.html) or [Opensearch](https://rubydoc.info/gems/opensearch-transport#configuration) for a complete list of options.
 
 ### Lograge
 
@@ -1575,11 +1682,12 @@ Reindex a subset of attributes to reduce time spent generating search data and c
 class Product < ApplicationRecord
   def search_data
     {
-      name: name
-    }.merge(search_prices)
+      name: name,
+      category: category
+    }.merge(prices_data)
   end
 
-  def search_prices
+  def prices_data
     {
       price: price,
       sale_price: sale_price
@@ -1591,68 +1699,7 @@ end
 And use:
 
 ```ruby
-Product.reindex(:search_prices)
-```
-
-### Performant Conversions
-
-Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
-
-```ruby
-class Product < ApplicationRecord
-  def search_data
-    {
-      name: name
-    }.merge(search_conversions)
-  end
-
-  def search_conversions
-    {
-      conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}
-    }
-  end
-end
-```
-
-Create a job to update the cache and reindex records with new conversions.
-
-```ruby
-class ReindexConversionsJob < ApplicationJob
-  def perform(class_name)
-    # get records that have a recent conversion
-    recently_converted_ids =
-      Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
-      .order(:convertable_id).distinct.pluck(:convertable_id)
-
-    # split into groups
-    recently_converted_ids.in_groups_of(1000, false) do |ids|
-      # fetch conversions
-      conversions =
-        Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
-        .group(:convertable_id, :query).distinct.count(:user_id)
-
-      # group conversions by record
-      conversions_by_record = {}
-      conversions.each do |(id, query), count|
-        (conversions_by_record[id] ||= {})[query] = count
-      end
-
-      # write to cache
-      conversions_by_record.each do |id, conversions|
-        Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
-      end
-
-      # partial reindex
-      class_name.constantize.where(id: ids).reindex(:search_conversions)
-    end
-  end
-end
-```
-
-Run the job with:
-
-```ruby
-ReindexConversionsJob.perform_later("Product")
+Product.reindex(:prices_data)
 ```
 
 ## Advanced
@@ -1798,6 +1845,87 @@ To query nested data, use dot notation.
 Product.search("san", fields: ["store.city"], where: {"store.zip_code" => 12345})
 ```
 
+## Nearest Neighbor Search
+
+*Available for Elasticsearch 8.6+ and OpenSearch 2.4+*
+
+```ruby
+class Product < ApplicationRecord
+  searchkick knn: {embedding: {dimensions: 3, distance: "cosine"}}
+end
+```
+
+Also supports `euclidean` and `inner_product`
+
+Reindex and search with:
+
+```ruby
+Product.search(knn: {field: :embedding, vector: [1, 2, 3]}, limit: 10)
+```
+
+## Semantic Search
+
+First, add [nearest neighbor search](#nearest-neighbor-search-unreleased-experimental) to your model
+
+```ruby
+class Product < ApplicationRecord
+  searchkick knn: {embedding: {dimensions: 768, distance: "cosine"}}
+end
+```
+
+Generate an embedding for each record (you can use an external service or a library like [Informers](https://github.com/ankane/informers))
+
+```ruby
+embed = Informers.pipeline("embedding", "Snowflake/snowflake-arctic-embed-m-v1.5")
+embed_options = {model_output: "sentence_embedding", pooling: "none"} # specific to embedding model
+
+Product.find_each do |product|
+  embedding = embed.(product.name, **embed_options)
+  product.update!(embedding: embedding)
+end
+```
+
+For search, generate an embedding for the query (the query prefix is specific to the [embedding model](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5))
+
+```ruby
+query_prefix = "Represent this sentence for searching relevant passages: "
+query_embedding = embed.(query_prefix + query, **embed_options)
+```
+
+And perform nearest neighbor search
+
+```ruby
+Product.search(knn: {field: :embedding, vector: query_embedding}, limit: 20)
+```
+
+See a [full example](examples/semantic.rb)
+
+## Hybrid Search
+
+Perform keyword search and semantic search in parallel
+
+```ruby
+keyword_search = Product.search(query, limit: 20)
+semantic_search = Product.search(knn: {field: :embedding, vector: query_embedding}, limit: 20)
+Searchkick.multi_search([keyword_search, semantic_search])
+```
+
+To combine the results, use Reciprocal Rank Fusion (RRF)
+
+```ruby
+Searchkick::Reranking.rrf(keyword_search, semantic_search).first(5)
+```
+
+Or a reranking model
+
+```ruby
+rerank = Informers.pipeline("reranking", "mixedbread-ai/mxbai-rerank-xsmall-v1")
+results = (keyword_search.to_a + semantic_search.to_a).uniq
+rerank.(query, results.map(&:name)).first(5).map { |v| results[v[:doc_id]] }
+```
+
+See a [full example](examples/hybrid.rb)
+
 ## Reference
 
 Reindex one record
@@ -2036,12 +2164,24 @@ Turn on misspellings after a certain number of characters
 Product.search("api", misspellings: {prefix_length: 2}) # api, apt, no ahi
 ```
 
-**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch
+**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch 1
 
 ```ruby
 Product.search("ah", misspellings: {prefix_length: 2}) # ah, no aha
 ```
 
+BigDecimal values are indexed as floats by default so they can be used for boosting. Convert them to strings to keep full precision.
+
+```ruby
+class Product < ApplicationRecord
+  def search_data
+    {
+      units: units.to_s("F")
+    }
+  end
+end
+```
+
 ## Gotchas
 
 ### Consistency

data/lib/searchkick/index.rb

@@ -67,7 +67,8 @@ module Searchkick
           index: name,
           body: {
             query: {match_all: {}},
-            size: 0
+            size: 0,
+            track_total_hits: true
           }
         )
 
@@ -98,7 +99,7 @@ module Searchkick
       record_data = RecordData.new(self, record).record_data
 
       # remove underscore
-      get_options = record_data.to_h { |k, v| [k.to_s.sub(/\A_/, "").to_sym, v] }
+      get_options = record_data.to_h { |k, v| [k.to_s.delete_prefix("_").to_sym, v] }
 
       client.get(get_options)["_source"]
     end
@@ -351,6 +352,8 @@ module Searchkick
     # http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/
     def full_reindex(relation, import: true, resume: false, retain: false, mode: nil, refresh_interval: nil, scope: nil, wait: nil)
       raise ArgumentError, "wait only available in :async mode" if !wait.nil? && mode != :async
+      # TODO raise ArgumentError in Searchkick 6
+      Searchkick.warn("Full reindex does not support :queue mode - use :async mode instead") if mode == :queue
 
       if resume
         index_name = all_indices.sort.last
@@ -418,7 +421,7 @@ module Searchkick
         true
       end
     rescue => e
-      if Searchkick.transport_error?(e) && e.message.include?("No handler for type [text]")
+      if Searchkick.transport_error?(e) && (e.message.include?("No handler for type [text]") || e.message.include?("class java.util.ArrayList cannot be cast to class java.util.Map"))
         raise UnsupportedVersionError
       end