searchkick 4.0.0 → 5.0.0

data/README.md

@@ -10,7 +10,7 @@ Searchkick handles:
 - special characters - `jalapeno` matches `jalapeño`
 - extra whitespace - `dishwasher` matches `dish washer`
 - misspellings - `zuchini` matches `zucchini`
-- custom synonyms - `qtip` matches `cotton swab`
+- custom synonyms - `pop` matches `soda`
 
 Plus:
 
@@ -20,40 +20,53 @@ Plus:
 - autocomplete
 - “Did you mean” suggestions
 - supports many languages
-- works with ActiveRecord, Mongoid, and NoBrainer
+- works with Active Record and Mongoid
+
+Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Autosuggest](https://github.com/ankane/autosuggest) for query suggestions
 
 :tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
 
-[![Build Status](https://travis-ci.org/ankane/searchkick.svg?branch=master)](https://travis-ci.org/ankane/searchkick)
+[![Build Status](https://github.com/ankane/searchkick/workflows/build/badge.svg?branch=master)](https://github.com/ankane/searchkick/actions)
 
 ## Contents
 
 - [Getting Started](#getting-started)
 - [Querying](#querying)
 - [Indexing](#indexing)
+- [Intelligent Search](#intelligent-search)
 - [Instant Search / Autocomplete](#instant-search--autocomplete)
 - [Aggregations](#aggregations)
+- [Testing](#testing)
 - [Deployment](#deployment)
 - [Performance](#performance)
-- [Elasticsearch DSL](#advanced)
+- [Advanced Search](#advanced)
 - [Reference](#reference)
+- [Contributing](#contributing)
+
+Searchkick 5.0 was recently released! See [how to upgrade](#upgrading)
 
 ## Getting Started
 
-[Install Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html). For Homebrew, use:
+Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html). For Homebrew, use:
 
 ```sh
 brew install elasticsearch
 brew services start elasticsearch
+# or
+brew install opensearch
+brew services start opensearch
 ```
 
-Add this line to your application’s Gemfile:
+Add these lines to your application’s Gemfile:
 
 ```ruby
-gem 'searchkick'
+gem "searchkick"
+
+gem "elasticsearch"   # select one
+gem "opensearch-ruby" # select one
 ```
 
-The latest version works with Elasticsearch 6 and 7. For Elasticsearch 5, use version 3.1.3 and [this readme](https://github.com/ankane/searchkick/blob/v3.1.3/README.md).
+The latest version works with Elasticsearch 7 and 8 and OpenSearch 1. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
 
 Add searchkick to models you want to search.
 
@@ -78,7 +91,7 @@ products.each do |product|
 end
 ```
 
-Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). As your search becomes more advanced, we recommend you use the [Elasticsearch DSL](#advanced) for maximum flexibility.
+Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch Search API](https://opensearch.org/docs/latest/opensearch/rest-api/search/). As your search becomes more advanced, we recommend you use the [search server DSL](#advanced) for maximum flexibility.
 
 ## Querying
 
@@ -98,14 +111,20 @@ Where
 
 ```ruby
 where: {
-  expires_at: {gt: Time.now}, # lt, gte, lte also available
-  orders_count: 1..10,        # equivalent to {gte: 1, lte: 10}
-  aisle_id: [25, 30],         # in
-  store_id: {not: 2},         # not
-  aisle_id: {not: [25, 30]},  # not in
-  user_ids: {all: [1, 3]},    # all elements in array
-  category: /frozen .+/,      # regexp
-  _or: [{in_stock: true}, {backordered: true}]
+  expires_at: {gt: Time.now},    # lt, gte, lte also available
+  orders_count: 1..10,           # equivalent to {gte: 1, lte: 10}
+  aisle_id: [25, 30],            # in
+  store_id: {not: 2},            # not
+  aisle_id: {not: [25, 30]},     # not in
+  user_ids: {all: [1, 3]},       # all elements in array
+  category: {like: "%frozen%"},  # like
+  category: {ilike: "%frozen%"}, # ilike
+  category: /frozen .+/,         # regexp
+  category: {prefix: "frozen"},  # prefix
+  store_id: {exists: true},      # exists
+  _or: [{in_stock: true}, {backordered: true}],
+  _and: [{in_stock: true}, {backordered: true}],
+  _not: {store_id: 1}            # negate a condition
 }
 ```
 
@@ -115,7 +134,7 @@ Order
 order: {_score: :desc} # most relevant first - default
 ```
 
-[All of these sort options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html)
+[All of these sort options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html)
 
 Limit / offset
 
@@ -129,11 +148,11 @@ Select
 select: [:name]
 ```
 
-[These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html)
+[These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#source-filtering)
 
 ### Results
 
-Searches return a `Searchkick::Results` object. This responds like an array to most methods.
+Searches return a `Searchkick::Relation` object. This responds like an array to most methods.
 
 ```ruby
 results = Product.search("milk")
@@ -142,7 +161,7 @@ results.any?
 results.each { |result| ... }
 ```
 
-By default, ids are fetched from Elasticsearch and records are fetched from your database. To fetch everything from Elasticsearch, use:
+By default, ids are fetched from the search server and records are fetched from your database. To fetch everything from the search server, use:
 
 ```ruby
 Product.search("apples", load: false)
@@ -160,12 +179,14 @@ Get the time the search took (in milliseconds)
 results.took
 ```
 
-Get the full response from Elasticsearch
+Get the full response from the search server
 
 ```ruby
 results.response
 ```
 
+**Note:** By default, Elasticsearch and OpenSearch [limit paging](#deep-paging) to the first 10,000 results for performance. This applies to the total count as well.
+
 ### Boosting
 
 Boost important fields
@@ -197,7 +218,7 @@ boost_by_recency: {created_at: {scale: "7d", decay: 0.5}}
 
 You can also boost by:
 
-- [Conversions](#keep-getting-better)
+- [Conversions](#intelligent-search)
 - [Distance](#boost-by-distance)
 
 ### Get Everything
@@ -287,7 +308,9 @@ To only match the exact order, use:
 User.search "fresh honey", match: :phrase
 ```
 
-### Language
+### Stemming and Language
+
+Searchkick stems words by default for better matching. `apple` and `apples` both stem to `appl`, so searches for either term will have the same matches.
 
 Searchkick defaults to English for stemming. To change this, use:
 
@@ -297,44 +320,95 @@ class Product < ApplicationRecord
 end
 ```
 
-[See the list of stemmers](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html)
-
-A few languages require plugins:
+See the [list of languages](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html#analysis-stemmer-tokenfilter-configure-parms). A few languages require plugins:
 
 - `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)
-- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/analysis-kuromoji.html)
+- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-smartcn.html)
+- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-kuromoji.html)
 - `korean` - [analysis-openkoreantext plugin](https://github.com/open-korean-text/elasticsearch-analysis-openkoreantext)
-- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/analysis-stempel.html)
-- `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/analysis-ukrainian.html)
+- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-nori.html)
+- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-stempel.html)
+- `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html)
 - `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)
 
-### Synonyms
+You can also use a Hunspell dictionary for stemming.
 
 ```ruby
 class Product < ApplicationRecord
-  searchkick synonyms: [["burger", "hamburger"], ["sneakers", "shoes"]]
+  searchkick stemmer: {type: "hunspell", locale: "en_US"}
 end
 ```
 
-Call `Product.reindex` after changing synonyms.
+Disable stemming with:
 
-Synonyms cannot be multiple words at the moment.
+```ruby
+class Image < ApplicationRecord
+  searchkick stem: false
+end
+```
+
+Exclude certain words from stemming with:
+
+```ruby
+class Image < ApplicationRecord
+  searchkick stem_exclusion: ["apples"]
+end
+```
+
+Or change how words are stemmed:
+
+```ruby
+class Image < ApplicationRecord
+  searchkick stemmer_override: ["apples => other"]
+end
+```
 
-To read synonyms from a file, use:
+### Synonyms
 
 ```ruby
-synonyms: -> { CSV.read("/some/path/synonyms.csv") }
+class Product < ApplicationRecord
+  searchkick search_synonyms: [["pop", "soda"], ["burger", "hamburger"]]
+end
 ```
 
+Call `Product.reindex` after changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.
+
 For directional synonyms, use:
 
 ```ruby
-synonyms: ["lightbulb => halogenlamp"]
+search_synonyms: ["lightbulb => halogenlamp"]
 ```
 
-### Tags and Dynamic Synonyms
+### Dynamic Synonyms
 
-The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms or tags without a full reindex. You can use a library like [ActsAsTaggableOn](https://github.com/mbleigh/acts-as-taggable-on) and do:
+The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
+
+#### Elasticsearch 7.3+ and OpenSearch
+
+For Elasticsearch 7.3+ and OpenSearch, we recommend placing synonyms in a file on the search server (in the `config` directory). This allows you to reload synonyms without reindexing.
+
+```txt
+pop, soda
+burger, hamburger
+```
+
+Then use:
+
+```ruby
+class Product < ApplicationRecord
+  searchkick search_synonyms: "synonyms.txt"
+end
+```
+
+And reload with:
+
+```ruby
+Product.search_index.reload_synonyms
+```
+
+#### Elasticsearch < 7.3
+
+You can use a library like [ActsAsTaggableOn](https://github.com/mbleigh/acts-as-taggable-on) and do:
 
 ```ruby
 class Product < ApplicationRecord
@@ -419,7 +493,7 @@ Search :ice_cream::cake: and get `ice cream cake`!
 Add this line to your application’s Gemfile:
 
 ```ruby
-gem 'gemoji-parser'
+gem "gemoji-parser"
 ```
 
 And use:
@@ -454,12 +528,10 @@ class Product < ApplicationRecord
 end
 ```
 
-By default, all records are indexed. To control which records are indexed, use the `should_index?` method together with the `search_import` scope.
+By default, all records are indexed. To control which records are indexed, use the `should_index?` method.
 
 ```ruby
 class Product < ApplicationRecord
-  scope :search_import, -> { where(active: true) }
-
   def should_index?
     active # only index active records
   end
@@ -486,7 +558,7 @@ For large data sets, try [parallel reindexing](#parallel-reindexing).
 
 - app starts
 
-### Stay Synced
+### Strategies
 
 There are four strategies for keeping the index synced with your database.
 
@@ -536,7 +608,7 @@ Searchkick.callbacks(false) do
 end
 ```
 
-#### Associations
+### Associations
 
 Data is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:
 
@@ -552,11 +624,31 @@ class Image < ApplicationRecord
 end
 ```
 
-### Analytics
+### Default Scopes
+
+If you have a default scope that filters records, use the `should_index?` method to exclude them from indexing:
+
+```ruby
+class Product < ApplicationRecord
+  default_scope { where(deleted_at: nil) }
+
+  def should_index?
+    deleted_at.nil?
+  end
+end
+```
+
+If you want to index and search filtered records, set:
 
-The best starting point to improve your search **by far** is to track searches and conversions.
+```ruby
+class Product < ApplicationRecord
+  searchkick unscope: true
+end
+```
 
-[Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
+## Intelligent Search
+
+The best starting point to improve your search **by far** is to track searches and conversions. [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
 
 ```ruby
 Product.search "apple", track: {user_id: current_user.id}
@@ -569,15 +661,9 @@ Focus on:
 - top searches with low conversions
 - top searches with no results
 
-### Keep Getting Better
-
-Searchkick can use conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
-
-The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.
+Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
 
-Searchkick automatically treats `apple` and `APPLE` the same.
-
-Next, add conversions to the index.
+Add conversion data with:
 
 ```ruby
 class Product < ApplicationRecord
@@ -588,7 +674,7 @@ class Product < ApplicationRecord
   def search_data
     {
       name: name,
-      conversions: searches.group(:query).uniq.count(:user_id)
+      conversions: searches.group(:query).distinct.count(:user_id)
       # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
     }
   end
@@ -601,9 +687,11 @@ Reindex and set up a cron job to add new conversions daily.
 rake searchkick:reindex CLASS=Product
 ```
 
-**Note:** For a more performant (but more advanced) approach, check out [performant conversions](#performant-conversions).
+This can make a huge difference on the quality of your search.
+
+For a more performant way to reindex conversion data, check out [performant conversions](#performant-conversions).
 
-### Personalized Results
+## Personalized Results
 
 Order results differently for each user. For example, show a user’s previously purchased products before other results.
 
@@ -624,13 +712,13 @@ Reindex and search with:
 Product.search "milk", boost_where: {orderer_ids: current_user.id}
 ```
 
-### Instant Search / Autocomplete
+## Instant Search / Autocomplete
 
 Autocomplete predicts what a user will type, making the search experience faster and easier.
 
 ![Autocomplete](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/autocomplete.png)
 
-**Note:** To autocomplete on general categories (like `cereal` rather than product names), check out [Autosuggest](https://github.com/ankane/autosuggest).
+**Note:** To autocomplete on search terms rather than results, check out [Autosuggest](https://github.com/ankane/autosuggest).
 
 **Note 2:** If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s *much* faster to load all records into JavaScript and autocomplete there (eliminates network requests).
 
@@ -692,7 +780,7 @@ Then add the search box and JavaScript code to a view.
 </script>
 ```
 
-### Suggestions
+## Suggestions
 
 ![Suggest](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/recursion.png)
 
@@ -709,7 +797,7 @@ products = Product.search "peantu butta", suggest: true
 products.suggestions # ["peanut butter"]
 ```
 
-### Aggregations
+## Aggregations
 
 [Aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) provide aggregated search data.
 
@@ -752,8 +840,6 @@ Order
 Product.search "wingtips", aggs: {color: {order: {"_key" => "asc"}}} # alphabetically
 ```
 
-**Note:** Use `_term` instead of `_key` in Elasticsearch 5
-
 [All of these options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order)
 
 Ranges
@@ -784,10 +870,10 @@ Product.search "pear", aggs: {products_per_year: {date_histogram: {field: :creat
 For other aggregation types, including sub-aggregations, use `body_options`:
 
 ```ruby
-Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}
+Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}}
 ```
 
-### Highlight
+## Highlight
 
 Specify which fields to index with highlighting.
 
@@ -838,9 +924,9 @@ Additional options can be specified for each field:
 Band.search "cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}}
 ```
 
-You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#_highlighted_fragments).
+You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html).
 
-### Similar Items
+## Similar Items
 
 Find similar items.
 
@@ -849,7 +935,7 @@ product = Product.first
 product.similar(fields: [:name], where: {size: "12 oz"})
 ```
 
-### Geospatial Searches
+## Geospatial Searches
 
 ```ruby
 class Restaurant < ApplicationRecord
@@ -889,7 +975,7 @@ Boost results by distance - closer results are boosted more
 Restaurant.search "noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}}
 ```
 
-Also supports [additional options](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#_decay_functions)
+Also supports [additional options](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay)
 
 ```ruby
 Restaurant.search "wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}}
@@ -985,13 +1071,13 @@ Product.search("soap", debug: true)
 
 This prints useful info to `stdout`.
 
-See how Elasticsearch scores your queries with:
+See how the search server scores your queries with:
 
 ```ruby
 Product.search("soap", explain: true).response
 ```
 
-See how Elasticsearch tokenizes your queries with:
+See how the search server tokenizes your queries with:
 
 ```ruby
 Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
@@ -1016,21 +1102,201 @@ Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
 
 See the [complete list of analyzers](https://github.com/ankane/searchkick/blob/31780ddac7a89eab1e0552a32b403f2040a37931/lib/searchkick/index_options.rb#L32).
 
+## Testing
+
+As you iterate on your search, it’s a good idea to add tests.
+
+For performance, only enable Searchkick callbacks for the tests that need it.
+
+### Parallel Tests
+
+Rails 6 enables parallel tests by default. Add to your `test/test_helper.rb`:
+
+```ruby
+class ActiveSupport::TestCase
+  parallelize_setup do |worker|
+    Searchkick.index_suffix = worker
+
+    # reindex models
+    Product.reindex
+
+    # and disable callbacks
+    Searchkick.disable_callbacks
+  end
+end
+```
+
+And use:
+
+```ruby
+class ProductTest < ActiveSupport::TestCase
+  def setup
+    Searchkick.enable_callbacks
+  end
+
+  def teardown
+    Searchkick.disable_callbacks
+  end
+
+  def test_search
+    Product.create!(name: "Apple")
+    Product.search_index.refresh
+    assert_equal ["Apple"], Product.search("apple").map(&:name)
+  end
+end
+```
+
+### Minitest
+
+Add to your `test/test_helper.rb`:
+
+```ruby
+# reindex models
+Product.reindex
+
+# and disable callbacks
+Searchkick.disable_callbacks
+```
+
+And use:
+
+```ruby
+class ProductTest < Minitest::Test
+  def setup
+    Searchkick.enable_callbacks
+  end
+
+  def teardown
+    Searchkick.disable_callbacks
+  end
+
+  def test_search
+    Product.create!(name: "Apple")
+    Product.search_index.refresh
+    assert_equal ["Apple"], Product.search("apple").map(&:name)
+  end
+end
+```
+
+### RSpec
+
+Add to your `spec/spec_helper.rb`:
+
+```ruby
+RSpec.configure do |config|
+  config.before(:suite) do
+    # reindex models
+    Product.reindex
+
+    # and disable callbacks
+    Searchkick.disable_callbacks
+  end
+
+  config.around(:each, search: true) do |example|
+    Searchkick.callbacks(nil) do
+      example.run
+    end
+  end
+end
+```
+
+And use:
+
+```ruby
+describe Product, search: true do
+  it "searches" do
+    Product.create!(name: "Apple")
+    Product.search_index.refresh
+    assert_equal ["Apple"], Product.search("apple").map(&:name)
+  end
+end
+```
+
+### Factory Bot
+
+Use a trait and an after `create` hook for each indexed model:
+
+```ruby
+FactoryBot.define do
+  factory :product do
+    # ...
+
+    # Note: This should be the last trait in the list so `reindex` is called
+    # after all the other callbacks complete.
+    trait :reindex do
+      after(:create) do |product, _evaluator|
+        product.reindex(refresh: true)
+      end
+    end
+  end
+end
+
+# use it
+FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
+```
+
+### GitHub Actions
+
+Check out [setup-elasticsearch](https://github.com/ankane/setup-elasticsearch) for an easy way to install Elasticsearch:
+
+```yml
+    - uses: ankane/setup-elasticsearch@v1
+```
+
+And [setup-opensearch](https://github.com/ankane/setup-opensearch) for an easy way to install OpenSearch:
+
+```yml
+    - uses: ankane/setup-opensearch@v1
+```
+
 ## Deployment
 
-Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the Elasticsearch server. This defaults to `http://localhost:9200`.
+For the search server, Searchkick uses `ENV["ELASTICSEARCH_URL"]` for Elasticsearch and `ENV["OPENSEARCH_URL"]` for OpenSearch. This defaults to `http://localhost:9200`.
+
+- [Elastic Cloud](#elastic-cloud)
+- [Heroku](#heroku)
+- [Amazon OpenSearch Service](#amazon-opensearch-service)
+- [Self-Hosted and Other](#self-hosted-and-other)
+
+### Elastic Cloud
+
+Create an initializer `config/initializers/elasticsearch.rb` with:
+
+```ruby
+ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
+```
+
+Then deploy and reindex:
+
+```sh
+rake searchkick:reindex:all
+```
 
 ### Heroku
 
-Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai) or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch). [SearchBox](https://elements.heroku.com/addons/searchbox) does not work at the moment.
+Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).
 
-For Bonsai:
+For Elasticsearch on Bonsai:
 
 ```sh
 heroku addons:create bonsai
 heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
 ```
 
+For OpenSearch on Bonsai:
+
+```sh
+heroku addons:create bonsai --engine=opensearch
+heroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`
+```
+
+For SearchBox:
+
+```sh
+heroku addons:create searchbox:starter
+heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`
+```
+
 For Elastic Cloud (previously Found):
 
 ```sh
@@ -1044,30 +1310,30 @@ Visit the Shield page and reset your password. You’ll need to add the username
 heroku config:get FOUNDELASTICSEARCH_URL
 ```
 
-And add `elastic:password@` right after `https://`:
+And add `elastic:password@` right after `https://` and add port `9243` at the end:
 
 ```sh
-heroku config:set ELASTICSEARCH_URL=https://elastic:password@12345.us-east-1.aws.found.io
+heroku config:set ELASTICSEARCH_URL=https://elastic:password@12345.us-east-1.aws.found.io:9243
 ```
 
 Then deploy and reindex:
 
 ```sh
-heroku run rake searchkick:reindex CLASS=Product
+heroku run rake searchkick:reindex:all
 ```
 
-### Amazon Elasticsearch Service
+### Amazon OpenSearch Service
 
-Create an initializer `config/initializers/elasticsearch.rb` with:
+Create an initializer `config/initializers/opensearch.rb` with:
 
 ```ruby
-ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com"
+ENV["OPENSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
 ```
 
 To use signed requests, include in your Gemfile:
 
 ```ruby
-gem 'faraday_middleware-aws-sigv4'
+gem "faraday_middleware-aws-sigv4"
 ```
 
 and add to your initializer:
@@ -1083,28 +1349,30 @@ Searchkick.aws_credentials = {
 Then deploy and reindex:
 
 ```sh
-rake searchkick:reindex CLASS=Product
+rake searchkick:reindex:all
 ```
 
-### Other
+### Self-Hosted and Other
 
-Create an initializer `config/initializers/elasticsearch.rb` with:
+Create an initializer with:
 
 ```ruby
-ENV["ELASTICSEARCH_URL"] = "https://user:password@host"
+ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
+# or
+ENV["OPENSEARCH_URL"] = "https://user:password@host:port"
 ```
 
 Then deploy and reindex:
 
 ```sh
-rake searchkick:reindex CLASS=Product
+rake searchkick:reindex:all
 ```
 
 ### Data Protection
 
-We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to Elasticsearch.
+We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to the search server.
 
-Bonsai, Elastic Cloud, and Amazon Elasticsearch all support encryption at rest and HTTPS.
+Bonsai, Elastic Cloud, and Amazon OpenSearch Service all support encryption at rest and HTTPS.
 
 ### Automatic Failover
 
@@ -1137,7 +1405,7 @@ See [Production Rails](https://github.com/ankane/production_rails) for other goo
 Significantly increase performance with faster JSON generation. Add [Oj](https://github.com/ohler55/oj) to your Gemfile.
 
 ```ruby
-gem 'oj'
+gem "oj"
 ```
 
 This speeds up all JSON generation and parsing in your application (automatically!)
@@ -1147,7 +1415,7 @@ This speeds up all JSON generation and parsing in your application (automaticall
 Significantly increase performance with persistent HTTP connections. Add [Typhoeus](https://github.com/typhoeus/typhoeus) to your Gemfile and it’ll automatically be used.
 
 ```ruby
-gem 'typhoeus'
+gem "typhoeus"
 ```
 
 To reduce log noise, create an initializer with:
@@ -1160,7 +1428,7 @@ If you run into issues on Windows, check out [this post](https://www.rastating.c
 
 ### Searchable Fields
 
-By default, all string fields are searchable (can be used in `fields` option). Speed up indexing and reduce index size by only making some fields searchable. This disables the `_all` field unless it’s listed.
+By default, all string fields are searchable (can be used in `fields` option). Speed up indexing and reduce index size by only making some fields searchable.
 
 ```ruby
 class Product < ApplicationRecord
@@ -1216,7 +1484,7 @@ Product.reindex(async: {wait: true})
 You can use [ActiveJob::TrafficControl](https://github.com/nickelser/activejob-traffic_control) to control concurrency. Install the gem:
 
 ```ruby
-gem 'activejob-traffic_control', '>= 0.1.3'
+gem "activejob-traffic_control", ">= 0.1.3"
 ```
 
 And create an initializer with:
@@ -1249,7 +1517,7 @@ Product.search_index.promote(index_name, update_refresh_interval: true)
 
 ### Queuing
 
-Push ids of records needing reindexed to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using [connection_pool](https://github.com/mperham/connection_pool).
+Push ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using [connection_pool](https://github.com/mperham/connection_pool).
 
 ```ruby
 Searchkick.redis = ConnectionPool.new { Redis.new }
@@ -1279,7 +1547,7 @@ For more tips, check out [Keeping Elasticsearch in Sync](https://www.elastic.co/
 
 ### Routing
 
-Searchkick supports [Elasticsearch’s routing feature](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
+Searchkick supports [routing](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
 
 ```ruby
 class Business < ApplicationRecord
@@ -1352,14 +1620,14 @@ class ReindexConversionsJob < ApplicationJob
     # get records that have a recent conversion
     recently_converted_ids =
       Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
-      .order(:convertable_id).uniq.pluck(:convertable_id)
+      .order(:convertable_id).distinct.pluck(:convertable_id)
 
     # split into groups
     recently_converted_ids.in_groups_of(1000, false) do |ids|
       # fetch conversions
       conversions =
         Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
-        .group(:convertable_id, :query).uniq.count(:user_id)
+        .group(:convertable_id, :query).distinct.count(:user_id)
 
       # group conversions by record
       conversions_by_record = {}
@@ -1387,7 +1655,7 @@ ReindexConversionsJob.perform_later("Product")
 
 ## Advanced
 
-Searchkick makes it easy to use the Elasticsearch DSL on its own.
+Searchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.
 
 ### Advanced Mapping
 
@@ -1396,10 +1664,8 @@ Create a custom mapping:
 ```ruby
 class Product < ApplicationRecord
   searchkick mappings: {
-    product: {
-      properties: {
-        name: {type: "keyword"}
-      }
+    properties: {
+      name: {type: "keyword"}
     }
   }
 end
@@ -1443,7 +1709,7 @@ products =
   end
 ```
 
-### Elasticsearch Gem
+### Client
 
 Searchkick is built on top of the [elasticsearch](https://github.com/elastic/elasticsearch-ruby) gem. To access the client directly, use:
 
@@ -1456,8 +1722,8 @@ Searchkick.client
 To batch search requests for performance, use:
 
 ```ruby
-products = Product.search("snacks", execute: false)
-coupons = Coupon.search("snacks", execute: false)
+products = Product.search("snacks")
+coupons = Coupon.search("snacks")
 Searchkick.multi_search([products, coupons])
 ```
 
@@ -1479,6 +1745,49 @@ Boost specific models with:
 indices_boost: {Category => 2, Product => 1}
 ```
 
+## Multi-Tenancy
+
+Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
+
+## Scroll API
+
+Searchkick also supports the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results). Scrolling is not intended for real time user requests, but rather for processing large amounts of data.
+
+```ruby
+Product.search("*", scroll: "1m").scroll do |batch|
+  # process batch ...
+end
+```
+
+You can also scroll batches manually.
+
+```ruby
+products = Product.search "*", scroll: "1m"
+while products.any?
+  # process batch ...
+
+  products = products.scroll
+end
+
+products.clear_scroll
+```
+
+## Deep Paging
+
+By default, Elasticsearch and OpenSearch limit paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:
+
+```ruby
+class Product < ApplicationRecord
+  searchkick deep_paging: true
+end
+```
+
+If you just need an accurate total count, you can instead use:
+
+```ruby
+Product.search("pears", body_options: {track_total_hits: true})
+```
+
 ## Nested Data
 
 To query nested data, use dot notation.
@@ -1570,7 +1879,7 @@ class Product < ApplicationRecord
   def search_data
     {
       name: name,
-      unique_user_conversions: searches.group(:query).uniq.count(:user_id),
+      unique_user_conversions: searches.group(:query).distinct.count(:user_id),
       # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
       total_conversions: searches.group(:query).count
       # {"ice cream" => 412, "chocolate" => 117, "cream" => 6}
@@ -1646,14 +1955,6 @@ class Product < ApplicationRecord
 end
 ```
 
-Turn off stemming
-
-```ruby
-class Product < ApplicationRecord
-  searchkick stem: false
-end
-```
-
 Turn on stemming for conversions
 
 ```ruby
@@ -1662,14 +1963,6 @@ class Product < ApplicationRecord
 end
 ```
 
-Use a different [similarity algorithm](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html) for scoring
-
-```ruby
-class Product < ApplicationRecord
-  searchkick similarity: "classic"
-end
-```
-
 Make search case-sensitive
 
 ```ruby
@@ -1704,14 +1997,7 @@ class Product < ApplicationRecord
 end
 ```
 
-Lazy searching
-
-```ruby
-products = Product.search("carrots", execute: false)
-products.each { ... } # search not executed until here
-```
-
-Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html), like `search_type` and `query_cache`
+Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-search-api-query-params) like `search_type`
 
 ```ruby
 Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})
@@ -1748,170 +2034,62 @@ Turn on misspellings after a certain number of characters
 Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi
 ```
 
-**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off
+**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch
 
 ```ruby
 Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
 ```
 
-## Testing
-
-For performance, only enable Searchkick callbacks for the tests that need it.
-
-### Minitest
-
-Add to your `test/test_helper.rb`:
-
-```ruby
-# reindex models
-Product.reindex
-
-# and disable callbacks
-Searchkick.disable_callbacks
-```
-
-And use:
-
-```ruby
-class ProductTest < Minitest::Test
-  def setup
-    Searchkick.enable_callbacks
-  end
-
-  def teardown
-    Searchkick.disable_callbacks
-  end
+## Gotchas
 
-  def test_search
-    Product.create!(name: "Apple")
-    Product.search_index.refresh
-    assert_equal ["Apple"], Product.search("apple").map(&:name)
-  end
-end
-```
-
-### RSpec
-
-Add to your `spec/spec_helper.rb`:
-
-```ruby
-RSpec.configure do |config|
-  config.before(:suite) do
-    # reindex models
-    Product.reindex
-
-    # and disable callbacks
-    Searchkick.disable_callbacks
-  end
-
-  config.around(:each, search: true) do |example|
-    Searchkick.callbacks(true) do
-      example.run
-    end
-  end
-end
-```
+### Consistency
 
-And use:
+Elasticsearch and OpenSearch are eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
 
 ```ruby
-describe Product, search: true do
-  it "searches" do
-    Product.create!(name: "Apple")
-    Product.search_index.refresh
-    assert_equal ["Apple"], Product.search("apple").map(&:name)
-  end
-end
+product.save!
+Product.search_index.refresh
 ```
 
-### Factory Bot
+### Inconsistent Scores
 
-Use a trait and an after `create` hook for each indexed model:
+Due to the distributed nature of Elasticsearch and OpenSearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
 
 ```ruby
-FactoryBot.define do
-  factory :product do
-    # ...
-
-    # Note: This should be the last trait in the list so `reindex` is called
-    # after all the other callbacks complete.
-    trait :reindex do
-      after(:create) do |product, _evaluator|
-        product.reindex(refresh: true)
-      end
-    end
-  end
+class Product < ApplicationRecord
+  searchkick settings: {number_of_shards: 1}
 end
-
-# use it
-FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
 ```
 
-### Parallel Tests
-
-Set:
-
-```ruby
-Searchkick.index_suffix = ENV["TEST_ENV_NUMBER"]
-```
-
-## Multi-Tenancy
-
-Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
+For convenience, this is set by default in the test environment.
 
 ## Upgrading
 
-See [how to upgrade to Searchkick 3](docs/Searchkick-3-Upgrade.md)
-
-## Elasticsearch 6 to 7 Upgrade
+### 5.0
 
-1. Install Searchkick 4
-2. Upgrade your Elasticsearch cluster
-
-## Elasticsearch 5 to 6 Upgrade
-
-Elasticsearch 6 removes the ability to reindex with the `_all` field. Before you upgrade, we recommend disabling this field manually and specifying default fields on your models.
+Searchkick 5 supports both the `elasticsearch` and `opensearch-ruby` gems. Add the one you want to use to your Gemfile:
 
 ```ruby
-class Product < ApplicationRecord
-  searchkick _all: false, default_fields: [:name]
-end
+gem "elasticsearch"
+# or
+gem "opensearch-ruby"
 ```
 
-If you need search across multiple fields, we recommend creating a similar field in your search data.
+If using the deprecated `faraday_middleware-aws-signers-v4` gem, switch to `faraday_middleware-aws-sigv4`.
 
-```ruby
-class Product < ApplicationRecord
-  def search_data
-    {
-      all: [name, size, quantity].join(" ")
-    }
-  end
-end
-```
-
-## Elasticsearch Gotchas
-
-### Consistency
-
-Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
+Also, searches now use lazy loading:
 
 ```ruby
-product.save!
-Product.search_index.refresh
-```
-
-### Inconsistent Scores
+# search not executed
+Product.search("milk")
 
-Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
-
-```ruby
-class Product < ApplicationRecord
-  searchkick settings: {number_of_shards: 1}
-end
+# search executed
+Product.search("milk").to_a
 ```
 
-For convenience, this is set by default in the test environment.
+And there’s a [new option](#default-scopes) for models with default scopes.
+
+Check out the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md) for the full list of changes.
 
 ## History
 
@@ -1930,13 +2108,13 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Write, clarify, or fix documentation
 - Suggest or add new features
 
-If you’re looking for ideas, [try here](https://github.com/ankane/searchkick/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22).
-
-To get started with development and testing:
+To get started with development:
 
 ```sh
 git clone https://github.com/ankane/searchkick.git
 cd searchkick
 bundle install
-rake test
+bundle exec rake test
 ```
+
+Feel free to open an issue to get feedback on your idea before spending too much time on it.