searchkick 4.1.1 → 4.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 79cd2e4afe02598dbdc951ed222c98ab51c8f6b0580c7beb70d7f6f5e9223c8f
-  data.tar.gz: 804ae10724df4b1f72fc93f157e185cd6acbc734952c64e1c596d20f26360ce9
+  metadata.gz: '0218ffe2be1e6fcebea56dfa7c8d112e2f43580d69b513d064f48c4905c51934'
+  data.tar.gz: 79b21178f3db14d84179aa3db7cbf5dae4d7adfa98a9a72e7f92991c9bff2ce0
 SHA512:
-  metadata.gz: 732852787481b5aaf69edda3850c50846d5d07a55ba73d41dbee864988f04c5480c8c1535af6b82374982da4f95de5735f36433e6a37b46a41053ecc975fc9ef
-  data.tar.gz: 6d4408714af66540fc2868d2d9d4bd7b72d468150f5a29c9200d4eb349ee89f3155c98536742eb0015f905de644345e832af3a752613007d424cbb2c353051c2
+  metadata.gz: ce5ec190b6d754210ce88f48197cc0c08a32b6d33cc9a904385613ae3ec5bd7490a1d6c711bcf6cd3645636d64fe0c7e91bc23e009a32416194c8fbbe0119956
+  data.tar.gz: 3b306480292cfbcdc5854cc491f3700ecfd88715d37f879aa5715f0782ac86f380aa4828bec2866d2ea74186e445b9d4380d4696c8c2abcf1404207afa8ede18

data/CHANGELOG.md

@@ -1,3 +1,29 @@
+## 4.4.0 (2020-06-17)
+
+- Added support for reloadable, multi-word, search time synonyms
+- Fixed another deprecation warning in Ruby 2.7
+
+## 4.3.1 (2020-05-13)
+
+- Fixed error with `exclude` in certain cases for Elasticsearch 7.7
+
+## 4.3.0 (2020-02-19)
+
+- Fixed `like` queries with `"` character
+- Better error when invalid parameters passed to `where`
+
+## 4.2.1 (2020-01-27)
+
+- Fixed deprecation warnings with Elasticsearch
+- Fixed deprecation warnings in Ruby 2.7
+
+## 4.2.0 (2019-12-18)
+
+- Added safety check for multiple `Model.reindex`
+- Added `deep_paging` option
+- Added request parameters to search notifications and curl representation
+- Removed curl from search notifications to prevent confusion
+
 ## 4.1.1 (2019-11-19)
 
 - Added `chinese2` and `korean2` languages

data/CONTRIBUTING.md

@@ -4,7 +4,7 @@ First, thanks for wanting to contribute. You’re awesome! :heart:
 
 ## Help
 
-We’re not able to provide support through GitHub Issues. If you’re looking for help with your code, try posting on [Stack Overflow](https://stackoverflow.com/).
+We’re not able to provide support through GitHub Issues. If you’re looking for help with your code, try posting on [Stack Overflow](https://stackoverflow.com/questions/tagged/searchkick).
 
 All features should be documented. If you don’t see a feature in the docs, assume it doesn’t exist.
 

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013-2019 Andrew Kane
+Copyright (c) 2013-2020 Andrew Kane
 
 MIT License
 

data/README.md

@@ -4,12 +4,6 @@
 
 **Searchkick learns what your users are looking for.** As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.
 
----
-
-[Searchkick Pro](https://searchkick.org/pro?utm_source=readme) is now available!
-
----
-
 Searchkick handles:
 
 - stemming - `tomatoes` matches `tomato`
@@ -49,7 +43,7 @@ Plus:
 
 ## Getting Started
 
-[Install Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html). For Homebrew, use:
+[Install Elasticsearch](https://www.elastic.co/downloads/elasticsearch). For Homebrew, use:
 
 ```sh
 brew install elasticsearch
@@ -137,15 +131,13 @@ Limit / offset
 limit: 20, offset: 40
 ```
 
-**Note:** By default, Elasticsearch [limits pagination](#deep-pagination) to the first 10,000 results for performance
-
 Select
 
 ```ruby
 select: [:name]
 ```
 
-[These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html)
+[These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-source-filtering)
 
 ### Results
 
@@ -182,6 +174,8 @@ Get the full response from Elasticsearch
 results.response
 ```
 
+**Note:** By default, Elasticsearch [limits paging](#deep-paging-master) to the first 10,000 results for performance. With Elasticsearch 7, this applies to the total count as well.
+
 ### Boosting
 
 Boost important fields
@@ -330,29 +324,52 @@ A few languages require plugins:
 
 ```ruby
 class Product < ApplicationRecord
-  searchkick synonyms: [["pop", "soda"], ["burger", "hamburger"]]
+  searchkick search_synonyms: [["pop", "soda"], ["burger", "hamburger"]]
 end
 ```
 
-Call `Product.reindex` after changing synonyms.
+Call `Product.reindex` after changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.
 
-Synonyms cannot be multiple words at the moment.
+For directional synonyms, use:
 
-To read synonyms from a file, use:
+```ruby
+synonyms: ["lightbulb => halogenlamp"]
+```
+
+### Dynamic Synonyms
+
+The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
+
+#### Elasticsearch 7.3+
+
+For Elasticsearch 7.3+, we recommend placing synonyms in a file on the Elasticsearch server (in the `config` directory). This allows you to reload synonyms without reindexing.
+
+```txt
+pop, soda
+burger, hamburger
+```
+
+Then use:
 
 ```ruby
-synonyms: -> { CSV.read("/some/path/synonyms.csv") }
+search_synonyms: "synonyms.txt"
 ```
 
-For directional synonyms, use:
+Add [elasticsearch-xpack](https://github.com/elastic/elasticsearch-ruby/tree/master/elasticsearch-xpack) to your Gemfile:
 
 ```ruby
-synonyms: ["lightbulb => halogenlamp"]
+gem 'elasticsearch-xpack', '>= 7.8.0.pre'
 ```
 
-### Tags and Dynamic Synonyms
+And use:
+
+```ruby
+Product.search_index.reload_synonyms
+```
 
-The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms or tags without a full reindex. You can use a library like [ActsAsTaggableOn](https://github.com/mbleigh/acts-as-taggable-on) and do:
+#### Elasticsearch < 7.3
+
+You can use a library like [ActsAsTaggableOn](https://github.com/mbleigh/acts-as-taggable-on) and do:
 
 ```ruby
 class Product < ApplicationRecord
@@ -554,7 +571,7 @@ Searchkick.callbacks(false) do
 end
 ```
 
-#### Associations
+### Associations
 
 Data is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:
 
@@ -770,8 +787,6 @@ Order
 Product.search "wingtips", aggs: {color: {order: {"_key" => "asc"}}} # alphabetically
 ```
 
-**Note:** Use `_term` instead of `_key` in Elasticsearch 5
-
 [All of these options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order)
 
 Ranges
@@ -1040,7 +1055,7 @@ Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the Elasticsearch server. This de
 
 ### Heroku
 
-Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai) or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch). [SearchBox](https://elements.heroku.com/addons/searchbox) does not work at the moment.
+Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).
 
 For Bonsai:
 
@@ -1049,6 +1064,13 @@ heroku addons:create bonsai
 heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
 ```
 
+For SearchBox:
+
+```sh
+heroku addons:create searchbox:starter
+heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`
+```
+
 For Elastic Cloud (previously Found):
 
 ```sh
@@ -1518,20 +1540,20 @@ end
 products.clear_scroll
 ```
 
-## Deep Pagination
+## Deep Paging
 
-By default, Elasticsearch limits pagination to the first 10,000 results [for performance](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you need to, you can use:
+By default, Elasticsearch limits paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:
 
 ```ruby
 class Product < ApplicationRecord
-  searchkick settings: {index: {max_result_window: 1000000000}}
+  searchkick deep_paging: true
 end
 ```
 
-And search with:
+If you just need an accurate total count with Elasticsearch 7, you can instead use:
 
 ```ruby
-Product.search("pears", limit: 1000000000, body_options: {track_total_hits: true})
+Product.search("pears", body_options: {track_total_hits: true})
 ```
 
 ## Nested Data
@@ -1813,6 +1835,44 @@ Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
 
 For performance, only enable Searchkick callbacks for the tests that need it.
 
+### Parallel Tests
+
+Rails 6 enables parallel tests by default. Add to your `test/test_helper.rb`:
+
+```ruby
+class ActiveSupport::TestCase
+  parallelize_setup do |worker|
+    Searchkick.index_suffix = worker
+
+    # reindex models
+    Product.reindex
+
+    # and disable callbacks
+    Searchkick.disable_callbacks
+  end
+end
+```
+
+And use:
+
+```ruby
+class ProductTest < ActiveSupport::TestCase
+  def setup
+    Searchkick.enable_callbacks
+  end
+
+  def teardown
+    Searchkick.disable_callbacks
+  end
+
+  def test_search
+    Product.create!(name: "Apple")
+    Product.search_index.refresh
+    assert_equal ["Apple"], Product.search("apple").map(&:name)
+  end
+end
+```
+
 ### Minitest
 
 Add to your `test/test_helper.rb`:
@@ -1860,7 +1920,7 @@ RSpec.configure do |config|
   end
 
   config.around(:each, search: true) do |example|
-    Searchkick.callbacks(true) do
+    Searchkick.callbacks(nil) do
       example.run
     end
   end
@@ -1902,14 +1962,6 @@ end
 FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
 ```
 
-### Parallel Tests
-
-Set:
-
-```ruby
-Searchkick.index_suffix = ENV["TEST_ENV_NUMBER"]
-```
-
 ## Multi-Tenancy
 
 Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
@@ -1985,13 +2037,11 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Write, clarify, or fix documentation
 - Suggest or add new features
 
-If you’re looking for ideas, [try here](https://github.com/ankane/searchkick/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22).
-
-To get started with development and testing:
+To get started with development:
 
 ```sh
 git clone https://github.com/ankane/searchkick.git
 cd searchkick
 bundle install
-rake test
+bundle exec rake test
 ```

data/lib/searchkick.rb

@@ -1,8 +1,10 @@
+# dependencies
 require "active_support"
 require "active_support/core_ext/hash/deep_merge"
 require "elasticsearch"
 require "hashie"
 
+# modules
 require "searchkick/bulk_indexer"
 require "searchkick/index"
 require "searchkick/indexer"
@@ -17,6 +19,7 @@ require "searchkick/record_indexer"
 require "searchkick/results"
 require "searchkick/version"
 
+# integrations
 require "searchkick/railtie" if defined?(Rails)
 require "searchkick/logging" if defined?(ActiveSupport::Notifications)
 
@@ -27,6 +30,7 @@ module Searchkick
   autoload :ProcessQueueJob, "searchkick/process_queue_job"
   autoload :ReindexV2Job,    "searchkick/reindex_v2_job"
 
+  # errors
   class Error < StandardError; end
   class MissingIndexError < Error; end
   class UnsupportedVersionError < Error; end
@@ -112,7 +116,7 @@ module Searchkick
     end
 
     options = options.merge(block: block) if block
-    query = Searchkick::Query.new(klass, term, options)
+    query = Searchkick::Query.new(klass, term, **options)
     if options[:execute] == false
       query
     else
@@ -142,7 +146,7 @@ module Searchkick
     end
   end
 
-  def self.callbacks(value)
+  def self.callbacks(value = nil)
     if block_given?
       previous_value = callbacks_value
       begin
@@ -249,6 +253,18 @@ module Searchkick
       }
     end
   end
+
+  # private
+  # methods are forwarded to base class
+  # this check to see if scope exists on that class
+  # it's a bit tricky, but this seems to work
+  def self.relation?(klass)
+    if klass.respond_to?(:current_scope)
+      !klass.current_scope.nil?
+    elsif defined?(Mongoid::Threaded)
+      !Mongoid::Threaded.current_scope(klass).nil?
+    end
+  end
 end
 
 # TODO find better ActiveModel hook

data/lib/searchkick/bulk_indexer.rb

@@ -141,7 +141,7 @@ module Searchkick
 
     def bulk_reindex_job(scope, batch_id, options)
       Searchkick.with_redis { |r| r.sadd(batches_key, batch_id) }
-      Searchkick::BulkReindexJob.perform_later({
+      Searchkick::BulkReindexJob.perform_later(**{
         class_name: scope.searchkick_options[:class_name],
         index_name: index.name,
         batch_id: batch_id

data/lib/searchkick/index.rb

@@ -47,7 +47,7 @@ module Searchkick
     end
 
     def refresh_interval
-      settings.values.first["settings"]["index"]["refresh_interval"]
+      index_settings["refresh_interval"]
     end
 
     def update_settings(settings)
@@ -174,6 +174,13 @@ module Searchkick
       Searchkick.search(like_text, model: record.class, **options)
     end
 
+    def reload_synonyms
+      require "elasticsearch/xpack"
+      raise Error, "Requires Elasticsearch 7.3+" if Searchkick.server_below?("7.3.0")
+      raise Error, "Requires elasticsearch-xpack 7.8+" unless client.xpack.respond_to?(:indices)
+      client.xpack.indices.reload_search_analyzers(index: name)
+    end
+
     # queue
 
     def reindex_queue
@@ -184,13 +191,20 @@ module Searchkick
 
     def reindex(relation, method_name, scoped:, full: false, scope: nil, **options)
       refresh = options.fetch(:refresh, !scoped)
+      options.delete(:refresh)
 
       if method_name
+        # TODO throw ArgumentError
+        Searchkick.warn("unsupported keywords: #{options.keys.map(&:inspect).join(", ")}") if options.any?
+
         # update
         import_scope(relation, method_name: method_name, scope: scope)
         self.refresh if refresh
         true
       elsif scoped && !full
+        # TODO throw ArgumentError
+        Searchkick.warn("unsupported keywords: #{options.keys.map(&:inspect).join(", ")}") if options.any?
+
         # reindex association
         import_scope(relation, scope: scope)
         self.refresh if refresh
@@ -249,6 +263,11 @@ module Searchkick
       end
     end
 
+    # private
+    def uuid
+      index_settings["uuid"]
+    end
+
     protected
 
     def client
@@ -259,6 +278,10 @@ module Searchkick
       @bulk_indexer ||= BulkIndexer.new(self)
     end
 
+    def index_settings
+      settings.values.first["settings"]["index"]
+    end
+
     def import_before_promotion(index, relation, **import_options)
       index.import_scope(relation, **import_options)
     end
@@ -285,6 +308,8 @@ module Searchkick
         scope: scope
       }
 
+      uuid = index.uuid
+
       # check if alias exists
       alias_exists = alias_exists?
       if alias_exists
@@ -292,6 +317,7 @@ module Searchkick
 
         # get existing indices to remove
         unless async
+          check_uuid(uuid, index.uuid)
           promote(index.name, update_refresh_interval: !refresh_interval.nil?)
           clean_indices unless retain
         end
@@ -316,6 +342,7 @@ module Searchkick
           # already promoted if alias didn't exist
           if alias_exists
             puts "Jobs complete. Promoting..."
+            check_uuid(uuid, index.uuid)
             promote(index.name, update_refresh_interval: !refresh_interval.nil?)
           end
           clean_indices unless retain
@@ -334,5 +361,15 @@ module Searchkick
 
       raise e
     end
+
+    # safety check
+    # still a chance for race condition since its called before promotion
+    # ideal is for user to disable automatic index creation
+    # https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation
+    def check_uuid(old_uuid, new_uuid)
+      if old_uuid != new_uuid
+        raise Searchkick::Error, "Safety check failed - only run one Model.reindex per model at a time"
+      end
+    end
   end
 end

data/lib/searchkick/index_options.rb

@@ -7,6 +7,7 @@ module Searchkick
 
       below62 = Searchkick.server_below?("6.2.0")
       below70 = Searchkick.server_below?("7.0.0")
+      below73 = Searchkick.server_below?("7.3.0")
 
       if below70
         index_type = options[:_type]
@@ -27,9 +28,6 @@ module Searchkick
         default_analyzer = :searchkick_index
         keyword_mapping = {type: "keyword"}
 
-        index_true_value = true
-        index_false_value = false
-
         keyword_mapping[:ignore_above] = options[:ignore_above] || 30000
 
         settings = {
@@ -126,12 +124,12 @@ module Searchkick
                 max_shingle_size: 5
               },
               searchkick_edge_ngram: {
-                type: "edgeNGram",
+                type: "edge_ngram",
                 min_gram: 1,
                 max_gram: 50
               },
               searchkick_ngram: {
-                type: "nGram",
+                type: "ngram",
                 min_gram: 1,
                 max_gram: 50
               },
@@ -288,9 +286,7 @@ module Searchkick
 
         # synonyms
         synonyms = options[:synonyms] || []
-
         synonyms = synonyms.call if synonyms.respond_to?(:call)
-
         if synonyms.any?
           settings[:analysis][:filter][:searchkick_synonym] = {
             type: "synonym",
@@ -313,6 +309,29 @@ module Searchkick
           end
         end
 
+        search_synonyms = options[:search_synonyms] || []
+        search_synonyms = search_synonyms.call if search_synonyms.respond_to?(:call)
+        if search_synonyms.is_a?(String) || search_synonyms.any?
+          if search_synonyms.is_a?(String)
+            synonym_graph = {
+              type: "synonym_graph",
+              synonyms_path: search_synonyms
+            }
+            synonym_graph[:updateable] = true unless below73
+          else
+            synonym_graph = {
+              type: "synonym_graph",
+              # TODO confirm this is correct
+              synonyms: search_synonyms.select { |s| s.size > 1 }.map { |s| s.is_a?(Array) ? s.join(",") : s }.map(&:downcase)
+            }
+          end
+          settings[:analysis][:filter][:searchkick_synonym_graph] = synonym_graph
+
+          [:searchkick_search2, :searchkick_word_search].each do |analyzer|
+            settings[:analysis][:analyzer][analyzer][:filter].insert(2, "searchkick_synonym_graph")
+          end
+        end
+
         if options[:wordnet]
           settings[:analysis][:filter][:searchkick_wordnet] = {
             type: "synonym",
@@ -356,13 +375,13 @@ module Searchkick
 
         mapping_options[:searchable].delete("_all")
 
-        analyzed_field_options = {type: default_type, index: index_true_value, analyzer: default_analyzer}
+        analyzed_field_options = {type: default_type, index: true, analyzer: default_analyzer}
 
         mapping_options.values.flatten.uniq.each do |field|
           fields = {}
 
           if options.key?(:filterable) && !mapping_options[:filterable].include?(field)
-            fields[field] = {type: default_type, index: index_false_value}
+            fields[field] = {type: default_type, index: false}
           else
             fields[field] = keyword_mapping
           end
@@ -378,7 +397,7 @@ module Searchkick
 
             mapping_options.except(:highlight, :searchable, :filterable, :word).each do |type, f|
               if options[:match] == type || f.include?(field)
-                fields[type] = {type: default_type, index: index_true_value, analyzer: "searchkick_#{type}_index"}
+                fields[type] = {type: default_type, index: true, analyzer: "searchkick_#{type}_index"}
               end
             end
           end
@@ -418,12 +437,12 @@ module Searchkick
         }
 
         if options.key?(:filterable)
-          dynamic_fields["{name}"] = {type: default_type, index: index_false_value}
+          dynamic_fields["{name}"] = {type: default_type, index: false}
         end
 
         unless options[:searchable]
           if options[:match] && options[:match] != :word
-            dynamic_fields[options[:match]] = {type: default_type, index: index_true_value, analyzer: "searchkick_#{options[:match]}_index"}
+            dynamic_fields[options[:match]] = {type: default_type, index: true, analyzer: "searchkick_#{options[:match]}_index"}
           end
 
           if word
@@ -456,6 +475,13 @@ module Searchkick
         mappings = mappings.symbolize_keys.deep_merge(custom_mapping.symbolize_keys)
       end
 
+      if options[:deep_paging]
+        if !settings.dig(:index, :max_result_window) && !settings[:"index.max_result_window"]
+          settings[:index] ||= {}
+          settings[:index][:max_result_window] = 1_000_000_000
+        end
+      end
+
       {
         settings: settings,
         mappings: mappings

data/lib/searchkick/logging.rb

@@ -132,7 +132,7 @@ module Searchkick
     def multi_search(searches)
       event = {
         name: "Multi Search",
-        body: searches.flat_map { |q| [q.params.except(:body).to_json, q.body.to_json] }.map { |v| "#{v}\n" }.join
+        body: searches.flat_map { |q| [q.params.except(:body).to_json, q.body.to_json] }.map { |v| "#{v}\n" }.join,
       }
       ActiveSupport::Notifications.instrument("multi_search.searchkick", event) do
         super
@@ -162,14 +162,17 @@ module Searchkick
 
       payload = event.payload
       name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
-      type = payload[:query][:type]
+
       index = payload[:query][:index].is_a?(Array) ? payload[:query][:index].join(",") : payload[:query][:index]
+      type = payload[:query][:type]
+      request_params = payload[:query].except(:index, :type, :body)
+
+      params = []
+      request_params.each do |k, v|
+        params << "#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}"
+      end
 
-      # no easy way to tell which host the client will use
-      host = Searchkick.client.transport.hosts.first
-      params = ["pretty"]
-      params << "scroll=#{payload[:query][:scroll]}" if payload[:query][:scroll]
-      debug "  #{color(name, YELLOW, true)}  curl #{host[:protocol]}://#{host[:host]}:#{host[:port]}/#{CGI.escape(index)}#{type ? "/#{type.map { |t| CGI.escape(t) }.join(',')}" : ''}/_search?#{params.join('&')} -H 'Content-Type: application/json' -d '#{payload[:query][:body].to_json}'"
+      debug "  #{color(name, YELLOW, true)}  #{index}#{type ? "/#{type.join(',')}" : ''}/_search#{params.any? ? '?' + params.join('&') : nil} #{payload[:query][:body].to_json}"
     end
 
     def request(event)
@@ -189,9 +192,7 @@ module Searchkick
       payload = event.payload
       name = "#{payload[:name]} (#{event.duration.round(1)}ms)"
 
-      # no easy way to tell which host the client will use
-      host = Searchkick.client.transport.hosts.first
-      debug "  #{color(name, YELLOW, true)}  curl #{host[:protocol]}://#{host[:host]}:#{host[:port]}/_msearch?pretty -H 'Content-Type: application/json' -d '#{payload[:body]}'"
+      debug "  #{color(name, YELLOW, true)}  _msearch #{payload[:body]}"
     end
   end
 

data/lib/searchkick/model.rb

@@ -3,9 +3,9 @@ module Searchkick
     def searchkick(**options)
       options = Searchkick.model_options.merge(options)
 
-      unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :default_fields,
+      unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :deep_paging, :default_fields,
         :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :language,
-        :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :settings, :similarity,
+        :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,
         :special_characters, :stem, :stem_conversions, :suggest, :synonyms, :text_end,
         :text_middle, :text_start, :word, :wordnet, :word_end, :word_middle, :word_start]
       raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?
@@ -41,7 +41,10 @@ module Searchkick
 
         class << self
           def searchkick_search(term = "*", **options, &block)
-            Searchkick.search(term, {model: self}.merge(options), &block)
+            # TODO throw error in next major version
+            Searchkick.warn("calling search on a relation is deprecated") if Searchkick.relation?(self)
+
+            Searchkick.search(term, model: self, **options, &block)
           end
           alias_method Searchkick.search_method_name, :searchkick_search if Searchkick.search_method_name
 
@@ -54,10 +57,11 @@ module Searchkick
           alias_method :search_index, :searchkick_index unless method_defined?(:search_index)
 
           def searchkick_reindex(method_name = nil, **options)
-            scoped = (respond_to?(:current_scope) && respond_to?(:default_scoped) && current_scope && current_scope.to_sql != default_scoped.to_sql) ||
+            # TODO relation = Searchkick.relation?(self)
+            relation = (respond_to?(:current_scope) && respond_to?(:default_scoped) && current_scope && current_scope.to_sql != default_scoped.to_sql) ||
               (respond_to?(:queryable) && queryable != unscoped.with_default_scope)
 
-            searchkick_index.reindex(searchkick_klass, method_name, scoped: scoped, **options)
+            searchkick_index.reindex(searchkick_klass, method_name, scoped: relation, **options)
           end
           alias_method :reindex, :searchkick_reindex unless method_defined?(:reindex)
 
@@ -79,8 +83,9 @@ module Searchkick
           RecordIndexer.new(self).reindex(method_name, **options)
         end unless method_defined?(:reindex)
 
+        # TODO switch to keyword arguments
         def similar(options = {})
-          self.class.searchkick_index.similar_record(self, options)
+          self.class.searchkick_index.similar_record(self, **options)
         end unless method_defined?(:similar)
 
         def search_data

data/lib/searchkick/query.rb

@@ -106,12 +106,15 @@ module Searchkick
       query = params
       type = query[:type]
       index = query[:index].is_a?(Array) ? query[:index].join(",") : query[:index]
+      request_params = query.except(:index, :type, :body)
 
       # no easy way to tell which host the client will use
       host = Searchkick.client.transport.hosts.first
       credentials = host[:user] || host[:password] ? "#{host[:user]}:#{host[:password]}@" : nil
       params = ["pretty"]
-      params << "scroll=#{options[:scroll]}" if options[:scroll]
+      request_params.each do |k, v|
+        params << "#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}"
+      end
       "curl #{host[:protocol]}://#{credentials}#{host[:host]}:#{host[:port]}/#{CGI.escape(index)}#{type ? "/#{type.map { |t| CGI.escape(t) }.join(',')}" : ''}/_search?#{params.join('&')} -H 'Content-Type: application/json' -d '#{query[:body].to_json}'"
     end
 
@@ -232,7 +235,9 @@ module Searchkick
 
       # pagination
       page = [options[:page].to_i, 1].max
-      per_page = (options[:limit] || options[:per_page] || 10_000).to_i
+      # maybe use index.max_result_window in the future
+      default_limit = searchkick_options[:deep_paging] ? 1_000_000_000 : 10_000
+      per_page = (options[:limit] || options[:per_page] || default_limit).to_i
       padding = [options[:padding].to_i, 0].max
       offset = options[:offset] || (page - 1) * per_page + padding
       scroll = options[:scroll]
@@ -345,7 +350,7 @@ module Searchkick
             field_misspellings = misspellings && (!misspellings_fields || misspellings_fields.include?(base_field(field)))
 
             if field == "_all" || field.end_with?(".analyzed")
-              shared_options[:cutoff_frequency] = 0.001 unless operator.to_s == "and" || field_misspellings == false
+              shared_options[:cutoff_frequency] = 0.001 unless operator.to_s == "and" || field_misspellings == false || (!below73? && !track_total_hits?)
               qs << shared_options.merge(analyzer: "searchkick_search")
 
               # searchkick_search and searchkick_search2 are the same for ukrainian
@@ -433,19 +438,20 @@ module Searchkick
 
         models = Array(options[:models])
         if models.any? { |m| m != m.searchkick_klass }
-          Searchkick.warn("Passing child models to models option throws off hits and pagination - use type option instead")
-
-          # TODO uncomment once aliases are supported with _index
-          # should be ES 7.5
+          # aliases are not supported with _index in ES below 7.5
           # see https://github.com/elastic/elasticsearch/pull/46640
-          # index_type_or =
-          #   models.map do |m|
-          #     v = {_index: m.searchkick_index.name}
-          #     v[:type] = m.searchkick_index.klass_document_type(m, true) if m != m.searchkick_klass
-          #     v
-          #   end
-
-          # where[:or] = Array(where[:or]) + [index_type_or]
+          if below75?
+            Searchkick.warn("Passing child models to models option throws off hits and pagination - use type option instead")
+          else
+            index_type_or =
+              models.map do |m|
+                v = {_index: m.searchkick_index.name}
+                v[:type] = m.searchkick_index.klass_document_type(m, true) if m != m.searchkick_klass
+                v
+              end
+
+            where[:or] = Array(where[:or]) + [index_type_or]
+          end
         end
 
         # start everything as efficient filters
@@ -516,6 +522,10 @@ module Searchkick
       # routing
       @routing = options[:routing] if options[:routing]
 
+      if track_total_hits?
+        payload[:track_total_hits] = true
+      end
+
       # merge more body options
       payload = payload.deep_merge(options[:body_options]) if options[:body_options]
 
@@ -564,7 +574,8 @@ module Searchkick
 
     def build_query(query, filters, should, must_not, custom_filters, multiply_filters)
       if filters.any? || must_not.any? || should.any?
-        bool = {must: query}
+        bool = {}
+        bool[:must] = query if query
         bool[:filter] = filters if filters.any?      # where
         bool[:must_not] = must_not if must_not.any?  # exclude
         bool[:should] = should if should.any?        # conversions
@@ -861,6 +872,11 @@ module Searchkick
     end
 
     def where_filters(where)
+      # if where.respond_to?(:permitted?) && !where.permitted?
+      #   # TODO check in more places
+      #   Searchkick.warn("Passing unpermitted parameters will raise an exception in Searchkick 5")
+      # end
+
       filters = []
       (where || {}).each do |field, value|
         field = :_id if field.to_s == "id"
@@ -943,10 +959,17 @@ module Searchkick
                 # % matches zero or more characters
                 # _ matches one character
                 # \ is escape character
-                regex = Regexp.escape(op_value).gsub(/(?<!\\)%/, ".*").gsub(/(?<!\\)_/, ".").gsub("\\%", "%").gsub("\\_", "_")
+                # escape Lucene reserved characters
+                # https://www.elastic.co/guide/en/elasticsearch/reference/current/regexp-syntax.html#regexp-optional-operators
+                reserved = %w(. ? + * | { } [ ] ( ) " \\)
+                regex = op_value.dup
+                reserved.each do |v|
+                  regex.gsub!(v, "\\" + v)
+                end
+                regex = regex.gsub(/(?<!\\)%/, ".*").gsub(/(?<!\\)_/, ".").gsub("\\%", "%").gsub("\\_", "_")
                 filters << {regexp: {field => {value: regex}}}
               when :prefix
-                filters << {prefix: {field => op_value}}
+                filters << {prefix: {field => {value: op_value}}}
               when :regexp # support for regexp queries without using a regexp ruby object
                 filters << {regexp: {field => {value: op_value}}}
               when :not, :_not # not equal
@@ -1026,7 +1049,16 @@ module Searchkick
 
         {regexp: {field => {value: source, flags: "NONE"}}}
       else
-        {term: {field => value}}
+        # TODO add this for other values
+        if value.as_json.is_a?(Enumerable)
+          # query will fail, but this is better
+          # same message as Active Record
+          # TODO make TypeError
+          # raise InvalidQueryError for backward compatibility
+          raise Searchkick::InvalidQueryError, "can't cast #{value.class.name}"
+        end
+
+        {term: {field => {value: value}}}
       end
     end
 
@@ -1090,6 +1122,14 @@ module Searchkick
       k.sub(/\.(analyzed|word_start|word_middle|word_end|text_start|text_middle|text_end|exact)\z/, "")
     end
 
+    def track_total_hits?
+      (searchkick_options[:deep_paging] && !below70?) || body_options[:track_total_hits]
+    end
+
+    def body_options
+      options[:body_options] || {}
+    end
+
     def below61?
       Searchkick.server_below?("6.1.0")
     end
@@ -1097,5 +1137,13 @@ module Searchkick
     def below70?
       Searchkick.server_below?("7.0.0")
     end
+
+    def below73?
+      Searchkick.server_below?("7.3.0")
+    end
+
+    def below75?
+      Searchkick.server_below?("7.5.0")
+    end
   end
 end

data/lib/searchkick/version.rb

@@ -1,3 +1,3 @@
 module Searchkick
-  VERSION = "4.1.1"
+  VERSION = "4.4.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: searchkick
 version: !ruby/object:Gem::Version
-  version: 4.1.1
+  version: 4.4.0
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-11-20 00:00:00.000000000 Z
+date: 2020-06-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activemodel
@@ -94,6 +94,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: elasticsearch-xpack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 7.8.0.pre
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 7.8.0.pre
 description: 
 email: andrew@chartkick.com
 executables: []
@@ -145,7 +159,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: Intelligent search made easy with Rails and Elasticsearch