searchkick 5.0.3 → 5.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec7741cb306a56f1a5ae5a07450c5c102236bc6103079fc34a5f602fa2853b31
-  data.tar.gz: 2bd747ee31846c901ce2a58125b34e9c90af6937a2c7dc041f2dd9f69701f1e9
+  metadata.gz: 8a93f95464f2b83d27f621d925326ea3b7728b2d71fff9831e20e07658ff0466
+  data.tar.gz: acf15b38474dea853f3e3574ea5fac21d4129920b6365e2c92ca58cfaf518cc0
 SHA512:
-  metadata.gz: f92f2a3c7bb27862b1768f5ecedc19ad0eab515d62515f94d72412ed7c22a20493049dd0e52cba21b3a43d8c846b0dc0c6cda40c68edc12f6af37e8487fb9403
-  data.tar.gz: f69a1cfc401bad0f09bda3f2788b1096cc004dcb765eb20b3d8ada06202185eeae630d953005298e5944ed1b8c239fa808eeebb32e9440cc7ae9a32e771469dc
+  metadata.gz: 51f9d3b6c110a7ec24988518b7e154fef368c2d027c00337e54096c3214871d29da9bdc9e549d4b9967742636bb8ae60d5a288ca066ff992691692a017b9bdda
+  data.tar.gz: 811825c1043231828c2d04a3a2081e453440fd470563680c8e470fea00569ffe57109bca64e07978ba8944d53a03f95c9fd4768872d6bf2ae6c4e87ede961e7a

data/CHANGELOG.md

@@ -1,3 +1,15 @@
+## 5.0.5 (2022-10-09)
+
+- Added `model` method to `Searchkick::Relation`
+- Fixed deprecation warning with `redis` gem
+- Fixed `respond_to?` method on relation loading relation
+- Fixed `Relation loaded` error for non-mutating methods on relation
+
+## 5.0.4 (2022-06-16)
+
+- Added `max_result_window` option
+- Improved error message for unsupported versions of Elasticsearch
+
 ## 5.0.3 (2022-03-13)
 
 - Fixed context for index name for inherited models

data/README.md

@@ -66,7 +66,7 @@ gem "elasticsearch"   # select one
 gem "opensearch-ruby" # select one
 ```
 
-The latest version works with Elasticsearch 7 and 8 and OpenSearch 1. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
+The latest version works with Elasticsearch 7 and 8 and OpenSearch 1 and 2. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
 
 Add searchkick to models you want to search.
 
@@ -292,12 +292,18 @@ Option | Matches | Example
 
 The default is `:word`. The most matches will happen with `:word_middle`.
 
+To specify different matching for different fields, use:
+
+```ruby
+Product.search(query, fields: [{name: :word_start}, {brand: :word_middle}])
+```
+
 ### Exact Matches
 
 To match a field exactly (case-sensitive), use:
 
 ```ruby
-Product.search(query, fields: [{email: :exact}, :name])
+Product.search(query, fields: [{name: :exact}])
 ```
 
 ### Phrase Matches
@@ -323,11 +329,11 @@ end
 See the [list of languages](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html#analysis-stemmer-tokenfilter-configure-parms). A few languages require plugins:
 
 - `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)
-- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-smartcn.html)
-- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-kuromoji.html)
+- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html)
+- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html)
 - `korean` - [analysis-openkoreantext plugin](https://github.com/open-korean-text/elasticsearch-analysis-openkoreantext)
-- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-nori.html)
-- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-stempel.html)
+- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-nori.html)
+- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-stempel.html)
 - `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html)
 - `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)
 
@@ -592,6 +598,14 @@ There are four strategies for keeping the index synced with your database.
   end
   ```
 
+  And reindex a record or relation manually.
+
+  ```ruby
+  product.reindex
+  # or
+  store.products.reindex(mode: :async)
+  ```
+
 You can also do bulk updates.
 
 ```ruby
@@ -608,6 +622,12 @@ Searchkick.callbacks(false) do
 end
 ```
 
+Or override the model’s strategy.
+
+```ruby
+product.reindex(mode: :async) # :inline or :queue
+```
+
 ### Associations
 
 Data is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:
@@ -654,20 +674,16 @@ The best starting point to improve your search **by far** is to track searches a
 Product.search("apple", track: {user_id: current_user.id})
 ```
 
-[See the docs](https://github.com/ankane/searchjoy) for how to install and use.
-
-Focus on:
-
-- top searches with low conversions
-- top searches with no results
+[See the docs](https://github.com/ankane/searchjoy) for how to install and use. Focus on top searches with a low conversion rate.
 
-Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
+Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches. This can make a huge difference on the quality of your search.
 
 Add conversion data with:
 
 ```ruby
 class Product < ApplicationRecord
-  has_many :searches, class_name: "Searchjoy::Search", as: :convertable
+  has_many :conversions, class_name: "Searchjoy::Conversion", as: :convertable
+  has_many :searches, class_name: "Searchjoy::Search", through: :conversions
 
   searchkick conversions: [:conversions] # name of field
 
@@ -681,15 +697,100 @@ class Product < ApplicationRecord
 end
 ```
 
-Reindex and set up a cron job to add new conversions daily.
+Reindex and set up a cron job to add new conversions daily. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
 
-```sh
-rake searchkick:reindex CLASS=Product
+### Performant Conversions
+
+A performant way to do conversions is to cache them to prevent N+1 queries. For Postgres, create a migration with:
+
+```ruby
+add_column :products, :search_conversions, :jsonb
+```
+
+For MySQL, use `:json`, and for others, use `:text` with a [JSON serializer](https://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Serialization/ClassMethods.html).
+
+Next, update your model. Create a separate method for conversion data so you can use [partial reindexing](#partial-reindexing).
+
+```ruby
+class Product < ApplicationRecord
+  searchkick conversions: [:conversions]
+
+  def search_data
+    {
+      name: name,
+      category: category
+    }.merge(conversions_data)
+  end
+
+  def conversions_data
+    {
+      conversions: search_conversions || {}
+    }
+  end
+end
+```
+
+Deploy and reindex your data. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
+
+```ruby
+Product.reindex
+```
+
+Then, create a job to update the conversions column and reindex records with new conversions. Here’s one you can use for Searchjoy:
+
+```ruby
+class UpdateConversionsJob < ApplicationJob
+  def perform(class_name, since: nil, update: true, reindex: true)
+    model = Searchkick.load_model(class_name)
+
+    # get records that have a recent conversion
+    recently_converted_ids =
+      Searchjoy::Conversion.where(convertable_type: class_name).where(created_at: since..)
+      .order(:convertable_id).distinct.pluck(:convertable_id)
+
+    # split into batches
+    recently_converted_ids.in_groups_of(1000, false) do |ids|
+      if update
+        # fetch conversions
+        conversions =
+          Searchjoy::Conversion.where(convertable_id: ids, convertable_type: class_name)
+          .joins(:search).where.not(searchjoy_searches: {user_id: nil})
+          .group(:convertable_id, :query).distinct.count(:user_id)
+
+        # group by record
+        conversions_by_record = {}
+        conversions.each do |(id, query), count|
+          (conversions_by_record[id] ||= {})[query] = count
+        end
+
+        # update conversions column
+        model.transaction do
+          conversions_by_record.each do |id, conversions|
+            model.where(id: id).update_all(search_conversions: conversions)
+          end
+        end
+      end
+
+      if reindex
+        # reindex conversions data
+        model.where(id: ids).reindex(:conversions_data)
+      end
+    end
+  end
+end
 ```
 
-This can make a huge difference on the quality of your search.
+Run the job:
 
-For a more performant way to reindex conversion data, check out [performant conversions](#performant-conversions).
+```ruby
+UpdateConversionsJob.perform_now("Product")
+```
+
+And set it up to run daily.
+
+```ruby
+UpdateConversionsJob.perform_later("Product", since: 1.day.ago)
+```
 
 ## Personalized Results
 
@@ -1575,11 +1676,12 @@ Reindex a subset of attributes to reduce time spent generating search data and c
 class Product < ApplicationRecord
   def search_data
     {
-      name: name
-    }.merge(search_prices)
+      name: name,
+      category: category
+    }.merge(prices_data)
   end
 
-  def search_prices
+  def prices_data
     {
       price: price,
       sale_price: sale_price
@@ -1591,68 +1693,7 @@ end
 And use:
 
 ```ruby
-Product.reindex(:search_prices)
-```
-
-### Performant Conversions
-
-Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
-
-```ruby
-class Product < ApplicationRecord
-  def search_data
-    {
-      name: name
-    }.merge(search_conversions)
-  end
-
-  def search_conversions
-    {
-      conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}
-    }
-  end
-end
-```
-
-Create a job to update the cache and reindex records with new conversions.
-
-```ruby
-class ReindexConversionsJob < ApplicationJob
-  def perform(class_name)
-    # get records that have a recent conversion
-    recently_converted_ids =
-      Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
-      .order(:convertable_id).distinct.pluck(:convertable_id)
-
-    # split into groups
-    recently_converted_ids.in_groups_of(1000, false) do |ids|
-      # fetch conversions
-      conversions =
-        Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
-        .group(:convertable_id, :query).distinct.count(:user_id)
-
-      # group conversions by record
-      conversions_by_record = {}
-      conversions.each do |(id, query), count|
-        (conversions_by_record[id] ||= {})[query] = count
-      end
-
-      # write to cache
-      conversions_by_record.each do |id, conversions|
-        Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
-      end
-
-      # partial reindex
-      class_name.constantize.where(id: ids).reindex(:search_conversions)
-    end
-  end
-end
-```
-
-Run the job with:
-
-```ruby
-ReindexConversionsJob.perform_later("Product")
+Product.reindex(:prices_data)
 ```
 
 ## Advanced
@@ -2036,12 +2077,24 @@ Turn on misspellings after a certain number of characters
 Product.search("api", misspellings: {prefix_length: 2}) # api, apt, no ahi
 ```
 
-**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch
+**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch 1
 
 ```ruby
 Product.search("ah", misspellings: {prefix_length: 2}) # ah, no aha
 ```
 
+BigDecimal values are indexed as floats by default so they can be used for boosting. Convert them to strings to keep full precision.
+
+```ruby
+class Product < ApplicationRecord
+  def search_data
+    {
+      units: units.to_s("F")
+    }
+  end
+end
+```
+
 ## Gotchas
 
 ### Consistency

data/lib/searchkick/index.rb

@@ -418,7 +418,7 @@ module Searchkick
         true
       end
     rescue => e
-      if Searchkick.transport_error?(e) && e.message.include?("No handler for type [text]")
+      if Searchkick.transport_error?(e) && (e.message.include?("No handler for type [text]") || e.message.include?("class java.util.ArrayList cannot be cast to class java.util.Map"))
         raise UnsupportedVersionError
       end
 

data/lib/searchkick/index_options.rb

@@ -19,7 +19,7 @@ module Searchkick
         mappings = generate_mappings.deep_symbolize_keys.deep_merge(custom_mappings)
       end
 
-      set_deep_paging(settings) if options[:deep_paging]
+      set_deep_paging(settings) if options[:deep_paging] || options[:max_result_window]
 
       {
         settings: settings,
@@ -525,7 +525,7 @@ module Searchkick
     def set_deep_paging(settings)
       if !settings.dig(:index, :max_result_window) && !settings[:"index.max_result_window"]
         settings[:index] ||= {}
-        settings[:index][:max_result_window] = 1_000_000_000
+        settings[:index][:max_result_window] = options[:max_result_window] || 1_000_000_000
       end
     end
 

data/lib/searchkick/model.rb

@@ -5,7 +5,7 @@ module Searchkick
 
       unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :deep_paging, :default_fields,
         :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :language,
-        :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,
+        :locations, :mappings, :match, :max_result_window, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,
         :special_characters, :stem, :stemmer, :stem_conversions, :stem_exclusion, :stemmer_override, :suggest, :synonyms, :text_end,
         :text_middle, :text_start, :unscope, :word, :word_end, :word_middle, :word_start]
       raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?

data/lib/searchkick/query.rb

@@ -254,6 +254,12 @@ module Searchkick
       offset = options[:offset] || (page - 1) * per_page + padding
       scroll = options[:scroll]
 
+      max_result_window = searchkick_options[:max_result_window]
+      if max_result_window
+        offset = max_result_window if offset > max_result_window
+        per_page = max_result_window - offset if offset + per_page > max_result_window
+      end
+
       # model and eager loading
       load = options[:load].nil? ? true : options[:load]
 

data/lib/searchkick/relation.rb

@@ -8,6 +8,9 @@ module Searchkick
     delegate :body, :params, to: :query
     delegate_missing_to :private_execute
 
+    attr_reader :model
+    alias_method :klass, :model
+
     def initialize(model, term = "*", **options)
       @model = model
       @term = term
@@ -26,8 +29,7 @@ module Searchkick
 
     def execute
       Searchkick.warn("The execute method is no longer needed")
-      private_execute
-      self
+      load
     end
 
     # experimental
@@ -195,10 +197,25 @@ module Searchkick
       Relation.new(@model, @term, **@options.except(*keys))
     end
 
+    # experimental
+    def load
+      private_execute
+      self
+    end
+
     def loaded?
       !@execute.nil?
     end
 
+    def respond_to_missing?(method_name, include_all)
+      Results.new(nil, nil, nil).respond_to?(method_name, include_all) || super
+    end
+
+    # TODO uncomment in 6.0
+    # def to_yaml
+    #   private_execute.to_a.to_yaml
+    # end
+
     private
 
     def private_execute
@@ -221,5 +238,10 @@ module Searchkick
     def ensure_permitted(obj)
       obj.to_h
     end
+
+    def initialize_copy(other)
+      super
+      @execute = nil
+    end
   end
 end

data/lib/searchkick/relation_indexer.rb

@@ -46,7 +46,7 @@ module Searchkick
     end
 
     def batch_completed(batch_id)
-      Searchkick.with_redis { |r| r.srem(batches_key, batch_id) }
+      Searchkick.with_redis { |r| r.srem(batches_key, [batch_id]) }
     end
 
     private
@@ -134,7 +134,7 @@ module Searchkick
     end
 
     def batch_job(class_name, batch_id, record_ids)
-      Searchkick.with_redis { |r| r.sadd(batches_key, batch_id) }
+      Searchkick.with_redis { |r| r.sadd(batches_key, [batch_id]) }
       Searchkick::BulkReindexJob.perform_later(
         class_name: class_name,
         index_name: index.name,

data/lib/searchkick/results.rb

@@ -3,6 +3,7 @@ module Searchkick
     include Enumerable
     extend Forwardable
 
+    # TODO remove klass and options in 6.0
     attr_reader :klass, :response, :options
 
     def_delegators :results, :each, :any?, :empty?, :size, :length, :slice, :[], :to_ary
@@ -13,6 +14,7 @@ module Searchkick
       @options = options
     end
 
+    # TODO make private in 6.0
     def results
       @results ||= with_hit.map(&:first)
     end

data/lib/searchkick/version.rb

@@ -1,3 +1,3 @@
 module Searchkick
-  VERSION = "5.0.3"
+  VERSION = "5.0.5"
 end

data/lib/searchkick.rb

@@ -135,8 +135,9 @@ module Searchkick
     @opensearch
   end
 
-  def self.server_below?(version)
-    server_version = opensearch? ? "7.10.2" : self.server_version
+  # TODO always check true version in Searchkick 6
+  def self.server_below?(version, true_version = false)
+    server_version = !true_version && opensearch? ? "7.10.2" : self.server_version
     Gem::Version.new(server_version.split("-")[0]) < Gem::Version.new(version.split("-")[0])
   end
 
@@ -284,7 +285,7 @@ module Searchkick
     relation
   end
 
-  # private
+  # public (for reindexing conversions)
   def self.load_model(class_name, allow_child: false)
     model = class_name.safe_constantize
     raise Error, "Could not find class: #{class_name}" unless model

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: searchkick
 version: !ruby/object:Gem::Version
-  version: 5.0.3
+  version: 5.0.5
 platform: ruby
 authors:
 - Andrew Kane
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-03-13 00:00:00.000000000 Z
+date: 2022-10-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activemodel