searchkick 5.0.3 → 5.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec7741cb306a56f1a5ae5a07450c5c102236bc6103079fc34a5f602fa2853b31
-  data.tar.gz: 2bd747ee31846c901ce2a58125b34e9c90af6937a2c7dc041f2dd9f69701f1e9
+  metadata.gz: 89c6a97d4c898be7f1f494cc4bfafc8aed5acc202a855588e81f73d86ea7b123
+  data.tar.gz: c362bb2916ec0b1fa83d72efd2314e747077b5cd696ed2ece089204be9452010
 SHA512:
-  metadata.gz: f92f2a3c7bb27862b1768f5ecedc19ad0eab515d62515f94d72412ed7c22a20493049dd0e52cba21b3a43d8c846b0dc0c6cda40c68edc12f6af37e8487fb9403
-  data.tar.gz: f69a1cfc401bad0f09bda3f2788b1096cc004dcb765eb20b3d8ada06202185eeae630d953005298e5944ed1b8c239fa808eeebb32e9440cc7ae9a32e771469dc
+  metadata.gz: bf1a9191ee97a19afde4c44b84c02b34b7f6b8b3ac464cdd34671dd60cb8a191ec338a028db8221c2a502fadf0059987f957c918221175631e8aa70344371ee7
+  data.tar.gz: 32e7b4b9f0899088e709a77389d8fa845ef68a3fd819d3823229eb1ffd99a6121647d5ab82d209e6254e4c242530d7698441d3b97f65a1e2436acbced6b6086f

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 5.0.4 (2022-06-16)
+
+- Added `max_result_window` option
+- Improved error message for unsupported versions of Elasticsearch
+
 ## 5.0.3 (2022-03-13)
 
 - Fixed context for index name for inherited models

data/README.md

@@ -66,7 +66,7 @@ gem "elasticsearch"   # select one
 gem "opensearch-ruby" # select one
 ```
 
-The latest version works with Elasticsearch 7 and 8 and OpenSearch 1. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
+The latest version works with Elasticsearch 7 and 8 and OpenSearch 1 and 2. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
 
 Add searchkick to models you want to search.
 
@@ -592,6 +592,14 @@ There are four strategies for keeping the index synced with your database.
   end
   ```
 
+  And reindex a record or relation manually.
+
+  ```ruby
+  product.reindex
+  # or
+  store.products.reindex(mode: :async)
+  ```
+
 You can also do bulk updates.
 
 ```ruby
@@ -608,6 +616,12 @@ Searchkick.callbacks(false) do
 end
 ```
 
+Or override the model’s strategy.
+
+```ruby
+product.reindex(mode: :async) # :inline or :queue
+```
+
 ### Associations
 
 Data is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:
@@ -654,20 +668,16 @@ The best starting point to improve your search **by far** is to track searches a
 Product.search("apple", track: {user_id: current_user.id})
 ```
 
-[See the docs](https://github.com/ankane/searchjoy) for how to install and use.
-
-Focus on:
-
-- top searches with low conversions
-- top searches with no results
+[See the docs](https://github.com/ankane/searchjoy) for how to install and use. Focus on top searches with a low conversion rate.
 
-Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
+Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches. This can make a huge difference on the quality of your search.
 
 Add conversion data with:
 
 ```ruby
 class Product < ApplicationRecord
-  has_many :searches, class_name: "Searchjoy::Search", as: :convertable
+  has_many :conversions, class_name: "Searchjoy::Conversion", as: :convertable
+  has_many :searches, class_name: "Searchjoy::Search", through: :conversions
 
   searchkick conversions: [:conversions] # name of field
 
@@ -681,15 +691,100 @@ class Product < ApplicationRecord
 end
 ```
 
-Reindex and set up a cron job to add new conversions daily.
+Reindex and set up a cron job to add new conversions daily. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
 
-```sh
-rake searchkick:reindex CLASS=Product
+### Performant Conversions
+
+A performant way to do conversions is to cache them to prevent N+1 queries. For Postgres, create a migration with:
+
+```ruby
+add_column :products, :search_conversions, :jsonb
+```
+
+For MySQL, use `:json`, and for others, use `:text` with a [JSON serializer](https://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Serialization/ClassMethods.html).
+
+Next, update your model. Create a separate method for conversion data so you can use [partial reindexing](#partial-reindexing).
+
+```ruby
+class Product < ApplicationRecord
+  searchkick conversions: [:conversions]
+
+  def search_data
+    {
+      name: name,
+      category: category
+    }.merge(conversions_data)
+  end
+
+  def conversions_data
+    {
+      conversions: search_conversions || {}
+    }
+  end
+end
+```
+
+Deploy and reindex your data. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
+
+```ruby
+Product.reindex
+```
+
+Then, create a job to update the conversions column and reindex records with new conversions. Here’s one you can use for Searchjoy:
+
+```ruby
+class UpdateConversionsJob < ApplicationJob
+  def perform(class_name, since: nil, update: true, reindex: true)
+    model = Searchkick.load_model(class_name)
+
+    # get records that have a recent conversion
+    recently_converted_ids =
+      Searchjoy::Conversion.where(convertable_type: class_name).where(created_at: since..)
+      .order(:convertable_id).distinct.pluck(:convertable_id)
+
+    # split into batches
+    recently_converted_ids.in_groups_of(1000, false) do |ids|
+      if update
+        # fetch conversions
+        conversions =
+          Searchjoy::Conversion.where(convertable_id: ids, convertable_type: class_name)
+          .joins(:search).where.not(searchjoy_searches: {user_id: nil})
+          .group(:convertable_id, :query).distinct.count(:user_id)
+
+        # group by record
+        conversions_by_record = {}
+        conversions.each do |(id, query), count|
+          (conversions_by_record[id] ||= {})[query] = count
+        end
+
+        # update conversions column
+        model.transaction do
+          conversions_by_record.each do |id, conversions|
+            model.where(id: id).update_all(search_conversions: conversions)
+          end
+        end
+      end
+
+      if reindex
+        # reindex conversions data
+        model.where(id: ids).reindex(:conversions_data)
+      end
+    end
+  end
+end
 ```
 
-This can make a huge difference on the quality of your search.
+Run the job:
 
-For a more performant way to reindex conversion data, check out [performant conversions](#performant-conversions).
+```ruby
+UpdateConversionsJob.perform_now("Product")
+```
+
+And set it up to run daily.
+
+```ruby
+UpdateConversionsJob.perform_later("Product", since: 1.day.ago)
+```
 
 ## Personalized Results
 
@@ -1575,11 +1670,12 @@ Reindex a subset of attributes to reduce time spent generating search data and c
 class Product < ApplicationRecord
   def search_data
     {
-      name: name
-    }.merge(search_prices)
+      name: name,
+      category: category
+    }.merge(prices_data)
   end
 
-  def search_prices
+  def prices_data
     {
       price: price,
       sale_price: sale_price
@@ -1591,68 +1687,7 @@ end
 And use:
 
 ```ruby
-Product.reindex(:search_prices)
-```
-
-### Performant Conversions
-
-Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
-
-```ruby
-class Product < ApplicationRecord
-  def search_data
-    {
-      name: name
-    }.merge(search_conversions)
-  end
-
-  def search_conversions
-    {
-      conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}
-    }
-  end
-end
-```
-
-Create a job to update the cache and reindex records with new conversions.
-
-```ruby
-class ReindexConversionsJob < ApplicationJob
-  def perform(class_name)
-    # get records that have a recent conversion
-    recently_converted_ids =
-      Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
-      .order(:convertable_id).distinct.pluck(:convertable_id)
-
-    # split into groups
-    recently_converted_ids.in_groups_of(1000, false) do |ids|
-      # fetch conversions
-      conversions =
-        Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
-        .group(:convertable_id, :query).distinct.count(:user_id)
-
-      # group conversions by record
-      conversions_by_record = {}
-      conversions.each do |(id, query), count|
-        (conversions_by_record[id] ||= {})[query] = count
-      end
-
-      # write to cache
-      conversions_by_record.each do |id, conversions|
-        Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
-      end
-
-      # partial reindex
-      class_name.constantize.where(id: ids).reindex(:search_conversions)
-    end
-  end
-end
-```
-
-Run the job with:
-
-```ruby
-ReindexConversionsJob.perform_later("Product")
+Product.reindex(:prices_data)
 ```
 
 ## Advanced
@@ -2036,12 +2071,24 @@ Turn on misspellings after a certain number of characters
 Product.search("api", misspellings: {prefix_length: 2}) # api, apt, no ahi
 ```
 
-**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch
+**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch 1
 
 ```ruby
 Product.search("ah", misspellings: {prefix_length: 2}) # ah, no aha
 ```
 
+BigDecimal values are indexed as floats by default so they can be used for boosting. Convert them to strings to keep full precision.
+
+```ruby
+class Product < ApplicationRecord
+  def search_data
+    {
+      units: units.to_s("F")
+    }
+  end
+end
+```
+
 ## Gotchas
 
 ### Consistency

data/lib/searchkick/index.rb

@@ -418,7 +418,7 @@ module Searchkick
         true
       end
     rescue => e
-      if Searchkick.transport_error?(e) && e.message.include?("No handler for type [text]")
+      if Searchkick.transport_error?(e) && (e.message.include?("No handler for type [text]") || e.message.include?("class java.util.ArrayList cannot be cast to class java.util.Map"))
         raise UnsupportedVersionError
       end
 

data/lib/searchkick/index_options.rb

@@ -19,7 +19,7 @@ module Searchkick
         mappings = generate_mappings.deep_symbolize_keys.deep_merge(custom_mappings)
       end
 
-      set_deep_paging(settings) if options[:deep_paging]
+      set_deep_paging(settings) if options[:deep_paging] || options[:max_result_window]
 
       {
         settings: settings,
@@ -525,7 +525,7 @@ module Searchkick
     def set_deep_paging(settings)
       if !settings.dig(:index, :max_result_window) && !settings[:"index.max_result_window"]
         settings[:index] ||= {}
-        settings[:index][:max_result_window] = 1_000_000_000
+        settings[:index][:max_result_window] = options[:max_result_window] || 1_000_000_000
       end
     end
 

data/lib/searchkick/model.rb

@@ -5,7 +5,7 @@ module Searchkick
 
       unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :deep_paging, :default_fields,
         :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :language,
-        :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,
+        :locations, :mappings, :match, :max_result_window, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,
         :special_characters, :stem, :stemmer, :stem_conversions, :stem_exclusion, :stemmer_override, :suggest, :synonyms, :text_end,
         :text_middle, :text_start, :unscope, :word, :word_end, :word_middle, :word_start]
       raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?

data/lib/searchkick/query.rb

@@ -254,6 +254,12 @@ module Searchkick
       offset = options[:offset] || (page - 1) * per_page + padding
       scroll = options[:scroll]
 
+      max_result_window = searchkick_options[:max_result_window]
+      if max_result_window
+        offset = max_result_window if offset > max_result_window
+        per_page = max_result_window - offset if offset + per_page > max_result_window
+      end
+
       # model and eager loading
       load = options[:load].nil? ? true : options[:load]
 

data/lib/searchkick/version.rb

@@ -1,3 +1,3 @@
 module Searchkick
-  VERSION = "5.0.3"
+  VERSION = "5.0.4"
 end

data/lib/searchkick.rb

@@ -135,8 +135,9 @@ module Searchkick
     @opensearch
   end
 
-  def self.server_below?(version)
-    server_version = opensearch? ? "7.10.2" : self.server_version
+  # TODO always check true version in Searchkick 6
+  def self.server_below?(version, true_version = false)
+    server_version = !true_version && opensearch? ? "7.10.2" : self.server_version
     Gem::Version.new(server_version.split("-")[0]) < Gem::Version.new(version.split("-")[0])
   end
 
@@ -284,7 +285,7 @@ module Searchkick
     relation
   end
 
-  # private
+  # public (for reindexing conversions)
   def self.load_model(class_name, allow_child: false)
     model = class_name.safe_constantize
     raise Error, "Could not find class: #{class_name}" unless model

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: searchkick
 version: !ruby/object:Gem::Version
-  version: 5.0.3
+  version: 5.0.4
 platform: ruby
 authors:
 - Andrew Kane
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-03-13 00:00:00.000000000 Z
+date: 2022-06-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activemodel