searchkick 4.4.0 → 4.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0218ffe2be1e6fcebea56dfa7c8d112e2f43580d69b513d064f48c4905c51934'
-  data.tar.gz: 79b21178f3db14d84179aa3db7cbf5dae4d7adfa98a9a72e7f92991c9bff2ce0
+  metadata.gz: ff49d45a4473f44b07e8f1654d47e9b0590fd71eb3c6567f62b2fbcdc1e837b9
+  data.tar.gz: fd6ac4c931c7250787bc6f905c6c72b8412355e6b505bb2ef9d8cf5731d2939d
 SHA512:
-  metadata.gz: ce5ec190b6d754210ce88f48197cc0c08a32b6d33cc9a904385613ae3ec5bd7490a1d6c711bcf6cd3645636d64fe0c7e91bc23e009a32416194c8fbbe0119956
-  data.tar.gz: 3b306480292cfbcdc5854cc491f3700ecfd88715d37f879aa5715f0782ac86f380aa4828bec2866d2ea74186e445b9d4380d4696c8c2abcf1404207afa8ede18
+  metadata.gz: 05630a20b302fd406e5545935861b6616db79e2dd12e2eb11fe82bb3733c8b21af1087b7ad8d0fd4bd93eb3a07d2bd67461815a293746a2d2c2d3f2148d7147b
+  data.tar.gz: 571528f3967fd921e92dda6413313d29bd9048ee832e44bdb1a31a297178c3ee2f0b31dead3c573b297da96b676fdd74467bd7aa514c6a97f2a38b2500901057

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+## 4.4.1 (2020-06-24)
+
+- Added `stem_exclusion` and `stemmer_override` options
+- Added `with_score` method to search results
+- Improved error message for `reload_synonyms` with non-OSS version of Elasticsearch
+- Improved output for reindex rake task
+
 ## 4.4.0 (2020-06-17)
 
 - Added support for reloadable, multi-word, search time synonyms
@@ -59,6 +66,7 @@ Breaking changes
 
 - Removed support for Elasticsearch 5
 - Removed support for multi-word synonyms (they no longer work with shingles)
+- Removed support for Active Record < 5
 
 ## 3.1.3 (2019-04-11)
 
@@ -125,7 +133,7 @@ Breaking changes
 Breaking changes
 
 - Removed support for Elasticsearch 2
-- Removed support for ActiveRecord < 4.2 and Mongoid < 5
+- Removed support for Active Record < 4.2 and Mongoid < 5
 - Types are no longer used
 - The `_all` field is disabled by default in Elasticsearch 5
 - Conversions are not stemmed by default
@@ -430,7 +438,7 @@ Breaking changes
 
 ## 0.8.3 (2014-09-20)
 
-- Added support for ActiveJob
+- Added support for Active Job
 - Added `timeout` setting
 - Fixed import with no records
 
@@ -537,7 +545,7 @@ Breaking changes
 
 ## 0.5.2 (2014-02-12)
 
-- Use after_commit hook for ActiveRecord to prevent data inconsistencies
+- Use after_commit hook for Active Record to prevent data inconsistencies
 
 ## 0.5.1 (2014-02-12)
 

data/README.md

@@ -33,13 +33,15 @@ Plus:
 - [Getting Started](#getting-started)
 - [Querying](#querying)
 - [Indexing](#indexing)
+- [Intelligent Search](#intelligent-search)
 - [Instant Search / Autocomplete](#instant-search--autocomplete)
 - [Aggregations](#aggregations)
+- [Testing](#testing)
 - [Deployment](#deployment)
 - [Performance](#performance)
 - [Elasticsearch DSL](#advanced)
 - [Reference](#reference)
-- [Testing](#testing)
+- [Contributing](#contributing)
 
 ## Getting Started
 
@@ -297,7 +299,9 @@ To only match the exact order, use:
 User.search "fresh honey", match: :phrase
 ```
 
-### Language
+### Stemming and Language
+
+Searchkick stems words by default for better matching. `apple` and `apples` both stem to `appl`, so searches for either term will have the same matches.
 
 Searchkick defaults to English for stemming. To change this, use:
 
@@ -307,9 +311,7 @@ class Product < ApplicationRecord
 end
 ```
 
-[See the list of stemmers](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html)
-
-A few languages require plugins:
+See the [list of stemmers](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html). A few languages require plugins:
 
 - `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)
 - `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-smartcn.html)
@@ -320,6 +322,30 @@ A few languages require plugins:
 - `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html)
 - `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)
 
+Disable stemming with:
+
+```ruby
+class Image < ApplicationRecord
+  searchkick stem: false
+end
+```
+
+Exclude certain words from stemming with:
+
+```ruby
+class Image < ApplicationRecord
+  searchkick stem_exclusion: ["apples"]
+end
+```
+
+Or change how words are stemmed:
+
+```ruby
+class Image < ApplicationRecord
+  searchkick stemmer_override: ["apples => other"]
+end
+```
+
 ### Synonyms
 
 ```ruby
@@ -333,7 +359,7 @@ Call `Product.reindex` after changing synonyms. Synonyms are applied at search t
 For directional synonyms, use:
 
 ```ruby
-synonyms: ["lightbulb => halogenlamp"]
+search_synonyms: ["lightbulb => halogenlamp"]
 ```
 
 ### Dynamic Synonyms
@@ -358,7 +384,7 @@ search_synonyms: "synonyms.txt"
 Add [elasticsearch-xpack](https://github.com/elastic/elasticsearch-ruby/tree/master/elasticsearch-xpack) to your Gemfile:
 
 ```ruby
-gem 'elasticsearch-xpack', '>= 7.8.0.pre'
+gem 'elasticsearch-xpack', '>= 7.8.0'
 ```
 
 And use:
@@ -521,7 +547,7 @@ For large data sets, try [parallel reindexing](#parallel-reindexing).
 
 - app starts
 
-### Stay Synced
+### Strategies
 
 There are four strategies for keeping the index synced with your database.
 
@@ -587,11 +613,9 @@ class Image < ApplicationRecord
 end
 ```
 
-### Analytics
-
-The best starting point to improve your search **by far** is to track searches and conversions.
+## Intelligent Search
 
-[Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
+The best starting point to improve your search **by far** is to track searches and conversions. [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
 
 ```ruby
 Product.search "apple", track: {user_id: current_user.id}
@@ -604,15 +628,9 @@ Focus on:
 - top searches with low conversions
 - top searches with no results
 
-### Keep Getting Better
-
-Searchkick can use conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
+Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
 
-The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.
-
-Searchkick automatically treats `apple` and `APPLE` the same.
-
-Next, add conversions to the index.
+Add conversion data with:
 
 ```ruby
 class Product < ApplicationRecord
@@ -636,9 +654,11 @@ Reindex and set up a cron job to add new conversions daily.
 rake searchkick:reindex CLASS=Product
 ```
 
-**Note:** For a more performant (but more advanced) approach, check out [performant conversions](#performant-conversions).
+This can make a huge difference on the quality of your search.
+
+For a more performant way to reindex conversion data, check out [performant conversions](#performant-conversions).
 
-### Personalized Results
+## Personalized Results
 
 Order results differently for each user. For example, show a user’s previously purchased products before other results.
 
@@ -659,7 +679,7 @@ Reindex and search with:
 Product.search "milk", boost_where: {orderer_ids: current_user.id}
 ```
 
-### Instant Search / Autocomplete
+## Instant Search / Autocomplete
 
 Autocomplete predicts what a user will type, making the search experience faster and easier.
 
@@ -727,7 +747,7 @@ Then add the search box and JavaScript code to a view.
 </script>
 ```
 
-### Suggestions
+## Suggestions
 
 ![Suggest](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/recursion.png)
 
@@ -744,7 +764,7 @@ products = Product.search "peantu butta", suggest: true
 products.suggestions # ["peanut butter"]
 ```
 
-### Aggregations
+## Aggregations
 
 [Aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) provide aggregated search data.
 
@@ -817,10 +837,10 @@ Product.search "pear", aggs: {products_per_year: {date_histogram: {field: :creat
 For other aggregation types, including sub-aggregations, use `body_options`:
 
 ```ruby
-Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}
+Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}}
 ```
 
-### Highlight
+## Highlight
 
 Specify which fields to index with highlighting.
 
@@ -873,7 +893,7 @@ Band.search "cinema", fields: [:name], highlight: {fields: {name: {fragment_size
 
 You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#_highlighted_fragments).
 
-### Similar Items
+## Similar Items
 
 Find similar items.
 
@@ -882,7 +902,7 @@ product = Product.first
 product.similar(fields: [:name], where: {size: "12 oz"})
 ```
 
-### Geospatial Searches
+## Geospatial Searches
 
 ```ruby
 class Restaurant < ApplicationRecord
@@ -1049,10 +1069,162 @@ Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
 
 See the [complete list of analyzers](https://github.com/ankane/searchkick/blob/31780ddac7a89eab1e0552a32b403f2040a37931/lib/searchkick/index_options.rb#L32).
 
+## Testing
+
+As you iterate on your search, it’s a good idea to add tests.
+
+For performance, only enable Searchkick callbacks for the tests that need it.
+
+### Parallel Tests
+
+Rails 6 enables parallel tests by default. Add to your `test/test_helper.rb`:
+
+```ruby
+class ActiveSupport::TestCase
+  parallelize_setup do |worker|
+    Searchkick.index_suffix = worker
+
+    # reindex models
+    Product.reindex
+
+    # and disable callbacks
+    Searchkick.disable_callbacks
+  end
+end
+```
+
+And use:
+
+```ruby
+class ProductTest < ActiveSupport::TestCase
+  def setup
+    Searchkick.enable_callbacks
+  end
+
+  def teardown
+    Searchkick.disable_callbacks
+  end
+
+  def test_search
+    Product.create!(name: "Apple")
+    Product.search_index.refresh
+    assert_equal ["Apple"], Product.search("apple").map(&:name)
+  end
+end
+```
+
+### Minitest
+
+Add to your `test/test_helper.rb`:
+
+```ruby
+# reindex models
+Product.reindex
+
+# and disable callbacks
+Searchkick.disable_callbacks
+```
+
+And use:
+
+```ruby
+class ProductTest < Minitest::Test
+  def setup
+    Searchkick.enable_callbacks
+  end
+
+  def teardown
+    Searchkick.disable_callbacks
+  end
+
+  def test_search
+    Product.create!(name: "Apple")
+    Product.search_index.refresh
+    assert_equal ["Apple"], Product.search("apple").map(&:name)
+  end
+end
+```
+
+### RSpec
+
+Add to your `spec/spec_helper.rb`:
+
+```ruby
+RSpec.configure do |config|
+  config.before(:suite) do
+    # reindex models
+    Product.reindex
+
+    # and disable callbacks
+    Searchkick.disable_callbacks
+  end
+
+  config.around(:each, search: true) do |example|
+    Searchkick.callbacks(nil) do
+      example.run
+    end
+  end
+end
+```
+
+And use:
+
+```ruby
+describe Product, search: true do
+  it "searches" do
+    Product.create!(name: "Apple")
+    Product.search_index.refresh
+    assert_equal ["Apple"], Product.search("apple").map(&:name)
+  end
+end
+```
+
+### Factory Bot
+
+Use a trait and an after `create` hook for each indexed model:
+
+```ruby
+FactoryBot.define do
+  factory :product do
+    # ...
+
+    # Note: This should be the last trait in the list so `reindex` is called
+    # after all the other callbacks complete.
+    trait :reindex do
+      after(:create) do |product, _evaluator|
+        product.reindex(refresh: true)
+      end
+    end
+  end
+end
+
+# use it
+FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
+```
+
 ## Deployment
 
 Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the Elasticsearch server. This defaults to `http://localhost:9200`.
 
+- [Elastic Cloud](#elastic-cloud)
+- [Heroku](#heroku)
+- [Amazon Elasticsearch Service](#amazon-elasticsearch-service)
+- [Self-Hosted and Other](#other)
+
+### Elastic Cloud
+
+Create an initializer `config/initializers/elasticsearch.rb` with:
+
+```ruby
+ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
+```
+
+Then deploy and reindex:
+
+```sh
+rake searchkick:reindex:all
+```
+
 ### Heroku
 
 Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).
@@ -1093,7 +1265,7 @@ heroku config:set ELASTICSEARCH_URL=https://elastic:password@12345.us-east-1.aws
 Then deploy and reindex:
 
 ```sh
-heroku run rake searchkick:reindex CLASS=Product
+heroku run rake searchkick:reindex:all
 ```
 
 ### Amazon Elasticsearch Service
@@ -1123,10 +1295,10 @@ Searchkick.aws_credentials = {
 Then deploy and reindex:
 
 ```sh
-rake searchkick:reindex CLASS=Product
+rake searchkick:reindex:all
 ```
 
-### Other
+### Self-Hosted and Other
 
 Create an initializer `config/initializers/elasticsearch.rb` with:
 
@@ -1137,7 +1309,7 @@ ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
 Then deploy and reindex:
 
 ```sh
-rake searchkick:reindex CLASS=Product
+rake searchkick:reindex:all
 ```
 
 ### Data Protection
@@ -1517,6 +1689,10 @@ Boost specific models with:
 indices_boost: {Category => 2, Product => 1}
 ```
 
+## Multi-Tenancy
+
+Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
+
 ## Scroll API
 
 Searchkick also supports the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html). Scrolling is not intended for real time user requests, but rather for processing large amounts of data.
@@ -1723,14 +1899,6 @@ class Product < ApplicationRecord
 end
 ```
 
-Turn off stemming
-
-```ruby
-class Product < ApplicationRecord
-  searchkick stem: false
-end
-```
-
 Turn on stemming for conversions
 
 ```ruby
@@ -1739,14 +1907,6 @@ class Product < ApplicationRecord
 end
 ```
 
-Use a different [similarity algorithm](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html) for scoring
-
-```ruby
-class Product < ApplicationRecord
-  searchkick similarity: "classic"
-end
-```
-
 Make search case-sensitive
 
 ```ruby
@@ -1831,172 +1991,11 @@ Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi
 Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
 ```
 
-## Testing
-
-For performance, only enable Searchkick callbacks for the tests that need it.
-
-### Parallel Tests
-
-Rails 6 enables parallel tests by default. Add to your `test/test_helper.rb`:
-
-```ruby
-class ActiveSupport::TestCase
-  parallelize_setup do |worker|
-    Searchkick.index_suffix = worker
-
-    # reindex models
-    Product.reindex
-
-    # and disable callbacks
-    Searchkick.disable_callbacks
-  end
-end
-```
-
-And use:
-
-```ruby
-class ProductTest < ActiveSupport::TestCase
-  def setup
-    Searchkick.enable_callbacks
-  end
-
-  def teardown
-    Searchkick.disable_callbacks
-  end
-
-  def test_search
-    Product.create!(name: "Apple")
-    Product.search_index.refresh
-    assert_equal ["Apple"], Product.search("apple").map(&:name)
-  end
-end
-```
-
-### Minitest
-
-Add to your `test/test_helper.rb`:
-
-```ruby
-# reindex models
-Product.reindex
-
-# and disable callbacks
-Searchkick.disable_callbacks
-```
-
-And use:
-
-```ruby
-class ProductTest < Minitest::Test
-  def setup
-    Searchkick.enable_callbacks
-  end
-
-  def teardown
-    Searchkick.disable_callbacks
-  end
-
-  def test_search
-    Product.create!(name: "Apple")
-    Product.search_index.refresh
-    assert_equal ["Apple"], Product.search("apple").map(&:name)
-  end
-end
-```
-
-### RSpec
-
-Add to your `spec/spec_helper.rb`:
-
-```ruby
-RSpec.configure do |config|
-  config.before(:suite) do
-    # reindex models
-    Product.reindex
-
-    # and disable callbacks
-    Searchkick.disable_callbacks
-  end
-
-  config.around(:each, search: true) do |example|
-    Searchkick.callbacks(nil) do
-      example.run
-    end
-  end
-end
-```
-
-And use:
-
-```ruby
-describe Product, search: true do
-  it "searches" do
-    Product.create!(name: "Apple")
-    Product.search_index.refresh
-    assert_equal ["Apple"], Product.search("apple").map(&:name)
-  end
-end
-```
-
-### Factory Bot
-
-Use a trait and an after `create` hook for each indexed model:
-
-```ruby
-FactoryBot.define do
-  factory :product do
-    # ...
-
-    # Note: This should be the last trait in the list so `reindex` is called
-    # after all the other callbacks complete.
-    trait :reindex do
-      after(:create) do |product, _evaluator|
-        product.reindex(refresh: true)
-      end
-    end
-  end
-end
-
-# use it
-FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
-```
-
-## Multi-Tenancy
-
-Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
-
-## Upgrading
-
-See [how to upgrade to Searchkick 3](docs/Searchkick-3-Upgrade.md)
-
 ## Elasticsearch 6 to 7 Upgrade
 
 1. Install Searchkick 4
 2. Upgrade your Elasticsearch cluster
 
-## Elasticsearch 5 to 6 Upgrade
-
-Elasticsearch 6 removes the ability to reindex with the `_all` field. Before you upgrade, we recommend disabling this field manually and specifying default fields on your models.
-
-```ruby
-class Product < ApplicationRecord
-  searchkick _all: false, default_fields: [:name]
-end
-```
-
-If you need search across multiple fields, we recommend creating a similar field in your search data.
-
-```ruby
-class Product < ApplicationRecord
-  def search_data
-    {
-      all: [name, size, quantity].join(" ")
-    }
-  end
-end
-```
-
 ## Elasticsearch Gotchas
 
 ### Consistency
@@ -2045,3 +2044,5 @@ cd searchkick
 bundle install
 bundle exec rake test
 ```
+
+Feel free to open an issue to get feedback on your idea before spending too much time on it.