searchkick 3.1.0 → 3.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2353bcf562a8f814430f473f0356e65cc8275c5bb21267fc46b96004c8edd27c
-  data.tar.gz: 5c570d53bc6d728a183de8ffb22056909ac748adebc6b436d2b8de785c3965fd
+  metadata.gz: b778411135d61db33ef8f82eb06990b41502d53cd5b2f4a21579f8c79b2f7aff
+  data.tar.gz: ba732ab1e72db7a9c5600bf99c3ae4b99d91cb47c97ad3ada0147888cb5e084f
 SHA512:
-  metadata.gz: c0aa2359ff46b280d0cc87fe57cf809760de414e3e40d7f5cb3c33cd43124ef3ad7aba824aa87f43390a188983ae2c1dfecba141ddaa19ab27a1cb50f6368447
-  data.tar.gz: bbf49af1a70667bdfa24ebee0227b0fa2b8d1738cfcbb860680d597a245d6860b1e5b56b978fc6b4aa9aa96e41bd20392e6e99116b7e69ff4a233a5480003709
+  metadata.gz: e4bb9b36b02989e29ed8adf179f45abf597ff91a63e23f442ab285f05523f15d8e00ff7a6fc634fca5553d6d91a05be467e589f8cf6e1b0de155b6db81c623fb
+  data.tar.gz: 417adb5009b35512b45c11024e4c9a413d713361a2bb0897b17e8c9128a788bcbb3b8df7801ef50a0fbae8269741e8a5f3c2405b4ddcdd0dab9f1ef914ef28bd

data/CHANGELOG.md

@@ -1,3 +1,12 @@
+## 3.1.1
+
+- Added per-field misspellings
+- Added `case_sensitive` option
+- Added `stem` option
+- Added `total_entries` option
+- Fixed `exclude` option with match all
+- Fixed `with_highlights` method
+
 ## 3.1.0
 
 - Added `:inline` as alias for `true` for `callbacks` and `mode` options

data/CONTRIBUTING.md

@@ -2,17 +2,15 @@
 
 First, thanks for wanting to contribute. You’re awesome! :heart:
 
-## Questions
+## Help
 
-Use [Stack Overflow](https://stackoverflow.com/) with the tag `searchkick`.
+We’re not able to provide support through GitHub Issues. If you’re looking for help with your code, try posting on [Stack Overflow](https://stackoverflow.com/).
 
-## Feature Requests
+All features should be documented. If you don’t see a feature in the docs, assume it doesn’t exist.
 
-Create an issue. Start the title with `[Idea]`.
+## Bugs
 
-## Issues
-
-Think you’ve discovered an issue?
+Think you’ve discovered a bug?
 
 1. Search existing issues to see if it’s been reported.
 2. Try the `master` branch to make sure it hasn’t been fixed.
@@ -32,6 +30,10 @@ If the above steps don’t help, create an issue.
 - Recreate the problem by forking [this gist](https://gist.github.com/ankane/f80b0923d9ae2c077f41997f7b704e5c). Include a link to your gist and the output in the issue.
 - For exceptions, include the complete backtrace.
 
+## New Features
+
+If you’d like to discuss a new feature, create an issue and start the title with `[Idea]`.
+
 ## Pull Requests
 
 Fork the project and create a pull request. A few tips:

data/README.md

@@ -271,15 +271,17 @@ Product.search "back", fields: [:name], match: :word_start
 
 Available options are:
 
-```ruby
-:word # default
-:word_start
-:word_middle
-:word_end
-:text_start
-:text_middle
-:text_end
-```
+Option | Matches | Example
+--- | --- | ---
+`:word` | entire word | `apple` matches `apple`
+`:word_start` | start of word | `app` matches `apple`
+`:word_middle` | any part of word | `ppl` matches `apple`
+`:word_end` | end of word | `ple` matches `apple`
+`:text_start` | start of text | `gre` matches `green apple`, `app` does not match
+`:text_middle` | any part of text | `een app` matches `green apple`
+`:text_end` | end of text | `ple` matches `green apple`, `een` does not match
+
+The default is `:word`. The most matches will happen with `:word_middle`.
 
 ### Exact Matches
 
@@ -410,6 +412,14 @@ Turn off misspellings with:
 Product.search "zuchini", misspellings: false # no zucchini
 ```
 
+Specify which fields can include misspellings with:
+
+```ruby
+Product.search "zucini", fields: [:name, :color], misspellings: {fields: [:name]}
+```
+
+> When doing this, you must also specify fields to search
+
 ### Bad Matches
 
 If a user searches `butter`, they may also get results for `peanut butter`. To prevent this, use:
@@ -645,7 +655,7 @@ Product.search "milk", boost_where: {orderer_ids: current_user.id}
 
 Autocomplete predicts what a user will type, making the search experience faster and easier.
 
-![Autocomplete](https://raw.githubusercontent.com/ankane/searchkick/gh-pages/autocomplete.png)
+![Autocomplete](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/autocomplete.png)
 
 **Note:** To autocomplete on general categories (like `cereal` rather than product names), check out [Autosuggest](https://github.com/ankane/autosuggest).
 
@@ -711,7 +721,7 @@ Then add the search box and JavaScript code to a view.
 
 ### Suggestions
 
-![Suggest](https://raw.githubusercontent.com/ankane/searchkick/gh-pages/recursion.png)
+![Suggest](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/recursion.png)
 
 ```ruby
 class Product < ApplicationRecord
@@ -730,7 +740,7 @@ products.suggestions # ["peanut butter"]
 
 [Aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) provide aggregated search data.
 
-![Aggregations](https://raw.githubusercontent.com/ankane/searchkick/gh-pages/facets.png)
+![Aggregations](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/facets.png)
 
 ```ruby
 products = Product.search "chuck taylor", aggs: [:product_type, :gender, :brand]
@@ -784,6 +794,12 @@ Minimum document count
 Product.search "apples", aggs: {store_id: {min_doc_count: 2}}
 ```
 
+Script support
+
+```ruby
+Product.search "*", aggs: {color: {script: {source: "'Color: ' + _value"}}}
+```
+
 Date histogram
 
 ```ruby
@@ -1045,7 +1061,7 @@ heroku addons:create bonsai
 heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
 ```
 
-For Found:
+For Elastic Cloud (previously Found):
 
 ```sh
 heroku addons:create foundelasticsearch
@@ -1078,7 +1094,7 @@ Create an initializer `config/initializers/elasticsearch.rb` with:
 ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com"
 ```
 
-To use signed request, include in your Gemfile:
+To use signed requests, include in your Gemfile:
 
 ```ruby
 gem 'faraday_middleware-aws-sigv4'
@@ -1105,7 +1121,7 @@ rake searchkick:reindex CLASS=Product
 Create an initializer `config/initializers/elasticsearch.rb` with:
 
 ```ruby
-ENV["ELASTICSEARCH_URL"] = "http://username:password@api.searchbox.io"
+ENV["ELASTICSEARCH_URL"] = "https://user:password@host"
 ```
 
 Then deploy and reindex:
@@ -1114,12 +1130,18 @@ Then deploy and reindex:
 rake searchkick:reindex CLASS=Product
 ```
 
+### Data Protection
+
+We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to Elasticsearch.
+
+Bonsai, Elastic Cloud, and Amazon Elasticsearch all support encryption at rest and HTTPS.
+
 ### Automatic Failover
 
 Create an initializer `config/initializers/elasticsearch.rb` with multiple hosts:
 
 ```ruby
-ENV["ELASTICSEARCH_URL"] = "http://localhost:9200,http://localhost:9201"
+ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
 ```
 
 See [elasticsearch-transport](https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-transport) for a complete list of options.
@@ -1218,7 +1240,7 @@ Searchkick.reindex_status(index_name)
 You can also have Searchkick wait for reindexing to complete
 
 ```ruby
-Searchkick.reindex(async: {wait: true})
+Product.reindex(async: {wait: true})
 ```
 
 You can use [ActiveJob::TrafficControl](https://github.com/nickelser/activejob-traffic_control) to control concurrency. Install the gem:
@@ -1427,7 +1449,7 @@ end
 And use the `body` option to search:
 
 ```ruby
-products = Product.search body: {match: {name: "milk"}}
+products = Product.search body: {query: {match: {name: "milk"}}}
 ```
 
 **Note:** This replaces the entire body, so other options are ignored.
@@ -1466,23 +1488,29 @@ Searchkick.client
 To batch search requests for performance, use:
 
 ```ruby
-fresh_products = Product.search("fresh", execute: false)
-frozen_products = Product.search("frozen", execute: false)
-Searchkick.multi_search([fresh_products, frozen_products])
+products = Product.search("snacks", execute: false)
+coupons = Coupon.search("snacks", execute: false)
+Searchkick.multi_search([products, coupons])
 ```
 
-Then use `fresh_products` and `frozen_products` as typical results.
+Then use `products` and `coupons` as typical results.
 
-**Note:** Errors are not raised as with single requests. Use the `error` method on each query to check for errors. Also, if you use the `below` option for misspellings, misspellings will be disabled.
+**Note:** Errors are not raised as with single requests. Use the `error` method on each query to check for errors.
 
 ## Multiple Indices
 
-Search across multiple indices with:
+Search across multiple models/indices with:
 
 ```ruby
 Searchkick.search "milk", index_name: [Product, Category]
 ```
 
+Specify conditions for different indices
+
+```ruby
+where: {_or: [{_type: "product", in_stock: true}, {_type: "category", active: true}]}
+```
+
 Boost specific indices with:
 
 ```ruby
@@ -1656,6 +1684,22 @@ class Product < ApplicationRecord
 end
 ```
 
+Turn off stemming
+
+```ruby
+class Product < ApplicationRecord
+  searchkick stem: false
+end
+```
+
+Turn on stemming for conversions
+
+```ruby
+class Product < ApplicationRecord
+  searchkick stem_conversions: true
+end
+```
+
 Use a different [similarity algorithm](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html) for scoring
 
 ```ruby
@@ -1664,6 +1708,16 @@ class Product < ApplicationRecord
 end
 ```
 
+Make search case-sensitive
+
+```ruby
+class Product < ApplicationRecord
+  searchkick case_sensitive: true
+end
+```
+
+**Note:** If misspellings are enabled (default), results with a single character case difference will match. Turn off misspellings if this is not desired.
+
 Change import batch size
 
 ```ruby
@@ -1691,6 +1745,14 @@ Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference
 Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})
 ```
 
+Set options across all models
+
+```ruby
+Searchkick.model_options = {
+  batch_size: 200
+}
+```
+
 Reindex conditionally
 
 ```ruby

data/lib/searchkick/bulk_indexer.rb

@@ -24,7 +24,7 @@ module Searchkick
           # to get the max _id without scripting since it's a string
 
           # TODO use primary key and prefix with table name
-          relation = relation.where("id > ?", total_docs)
+          relation = relation.where("id > ?", index.total_docs)
         end
 
         relation = relation.select("id").except(:includes, :preload) if async

data/lib/searchkick/index_options.rb

@@ -225,6 +225,18 @@ module Searchkick
           }
         end
 
+        if options[:case_sensitive]
+          settings[:analysis][:analyzer].each do |_, analyzer|
+            analyzer[:filter].delete("lowercase")
+          end
+        end
+
+        if options[:stem] == false
+          settings[:analysis][:analyzer].each do |_, analyzer|
+            analyzer[:filter].delete("searchkick_stemmer")
+          end
+        end
+
         settings = settings.symbolize_keys.deep_merge((options[:settings] || {}).symbolize_keys)
 
         # synonyms

data/lib/searchkick/model.rb

@@ -3,10 +3,10 @@ module Searchkick
     def searchkick(**options)
       options = Searchkick.model_options.merge(options)
 
-      unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :conversions, :default_fields,
+      unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :default_fields,
         :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :language,
         :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :settings, :similarity,
-        :special_characters, :stem_conversions, :suggest, :synonyms, :text_end,
+        :special_characters, :stem, :stem_conversions, :suggest, :synonyms, :text_end,
         :text_middle, :text_start, :word, :wordnet, :word_end, :word_middle, :word_start]
       raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?
 

data/lib/searchkick/query.rb

@@ -19,7 +19,7 @@ module Searchkick
         :boost_by, :boost_by_distance, :boost_by_recency, :boost_where, :conversions, :conversions_term, :debug, :emoji, :exclude, :execute, :explain,
         :fields, :highlight, :includes, :index_name, :indices_boost, :limit, :load,
         :match, :misspellings, :model_includes, :offset, :operator, :order, :padding, :page, :per_page, :profile,
-        :request_params, :routing, :scope_results, :select, :similar, :smart_aggs, :suggest, :track, :type, :where]
+        :request_params, :routing, :scope_results, :select, :similar, :smart_aggs, :suggest, :total_entries, :track, :type, :where]
       raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?
 
       term = term.to_s
@@ -116,7 +116,8 @@ module Searchkick
         misspellings: @misspellings,
         term: term,
         scope_results: options[:scope_results],
-        index_name: options[:index_name]
+        index_name: options[:index_name],
+        total_entries: options[:total_entries]
       }
 
       if options[:debug]
@@ -250,7 +251,7 @@ module Searchkick
           if fields != ["_all"]
             query[:more_like_this][:fields] = fields
           end
-        elsif all
+        elsif all && !options[:exclude]
           query = {
             match_all: {}
           }
@@ -280,6 +281,15 @@ module Searchkick
             prefix_length = (misspellings.is_a?(Hash) && misspellings[:prefix_length]) || 0
             default_max_expansions = @misspellings_below ? 20 : 3
             max_expansions = (misspellings.is_a?(Hash) && misspellings[:max_expansions]) || default_max_expansions
+            misspellings_fields = misspellings.is_a?(Hash) && misspellings.key?(:fields) && misspellings[:fields].map { |f| "#{f}.#{@match_suffix}" }
+
+            if misspellings_fields
+              missing_fields = misspellings_fields - fields
+              if missing_fields.any?
+                raise ArgumentError, "All fields in per-field misspellings must also be specified in fields option"
+              end
+            end
+
             @misspellings = true
           else
             @misspellings = false
@@ -314,8 +324,10 @@ module Searchkick
             exclude_analyzer = nil
             exclude_field = field
 
+            field_misspellings = misspellings && (!misspellings_fields || misspellings_fields.include?(field))
+
             if field == "_all" || field.end_with?(".analyzed")
-              shared_options[:cutoff_frequency] = 0.001 unless operator.to_s == "and" || misspellings == false
+              shared_options[:cutoff_frequency] = 0.001 unless operator.to_s == "and" || field_misspellings == false
               qs << shared_options.merge(analyzer: "searchkick_search")
 
               # searchkick_search and searchkick_search2 are the same for ukrainian
@@ -334,7 +346,7 @@ module Searchkick
               exclude_analyzer = analyzer
             end
 
-            if misspellings != false && match_type == :match
+            if field_misspellings != false && match_type == :match
               qs.concat(qs.map { |q| q.except(:cutoff_frequency).merge(fuzziness: edit_distance, prefix_length: prefix_length, max_expansions: max_expansions, boost: factor).merge(transpositions) })
             end
 
@@ -372,13 +384,22 @@ module Searchkick
             end
           end
 
-          payload = {
-            dis_max: {
-              queries: queries
+          # all + exclude option
+          if all
+            query = {
+              match_all: {}
+            }
+
+            should = []
+          else
+            payload = {
+              dis_max: {
+                queries: queries
+              }
             }
-          }
 
-          should.concat(set_conversions)
+            should.concat(set_conversions)
+          end
 
           query = payload
         end
@@ -487,9 +508,7 @@ module Searchkick
           ["_all"]
         elsif all && default_match == :phrase
           ["_all.phrase"]
-        elsif term == "*"
-          []
-        elsif default_match == :exact
+        elsif term != "*" && default_match == :exact
           raise ArgumentError, "Must specify fields to search"
         else
           [default_match == :word ? "*.analyzed" : "*.#{default_match}"]
@@ -718,7 +737,7 @@ module Searchkick
       aggs = Hash[aggs.map { |f| [f, {}] }] if aggs.is_a?(Array) # convert to more advanced syntax
       aggs.each do |field, agg_options|
         size = agg_options[:limit] ? agg_options[:limit] : 1_000
-        shared_agg_options = agg_options.slice(:order, :min_doc_count)
+        shared_agg_options = agg_options.except(:limit, :field, :ranges, :date_ranges, :where)
 
         if agg_options[:ranges]
           payload[:aggs][field] = {
@@ -735,19 +754,15 @@ module Searchkick
             }.merge(shared_agg_options)
           }
         elsif (histogram = agg_options[:date_histogram])
-          interval = histogram[:interval]
           payload[:aggs][field] = {
-            date_histogram: {
-              field: histogram[:field],
-              interval: interval
-            }
-          }
+            date_histogram: histogram
+          }.merge(shared_agg_options)
         elsif (metric = @@metric_aggs.find { |k| agg_options.has_key?(k) })
           payload[:aggs][field] = {
             metric => {
               field: agg_options[metric][:field] || field
             }
-          }
+          }.merge(shared_agg_options)
         else
           payload[:aggs][field] = {
             terms: {