searchkick 4.6.3 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6937cc5456846216a4ea52c2274dfc43220e7f848dde276f0102ce48de3184a6
-  data.tar.gz: 0bfcc6a4e5a893629f9cac734498efba357dbd19e6641c21af7ea8e82f94f1ba
+  metadata.gz: bc223f746f73c44e5c6f56b5e351cb31ae5a655069d92f5bfd2a0b2e531afae8
+  data.tar.gz: 1969964f34a5adaf3ed746230fbef57673454f7304d890466e2814d27eb2cb47
 SHA512:
-  metadata.gz: 0e7e41267464a8d32641edf6a578a6c65a97170f40e2f129effc8ff1da4144c30f00b88e52ddf6c51ffba366ec3cdfb263890e1539165ad47581c1723bf6b992
-  data.tar.gz: 49511b973a2fb7dce8d7952c0e5ccd676db357d2d965f46321e1b56bd5213f9f0c44f461f0010344f585e799dcbb13d9c4482772e20ab996462be86430c040cf
+  metadata.gz: 17259c3e071d69c3852584aba480b523211b167f1249951f32d1bd2f761fc07322144283f0ccabac4878c0a2ff2f4c9dd8def2db2beac004c449839052e503ea
+  data.tar.gz: 1109a53d99eb30f7e76cab8df00505c82af5e898c4e3f1821773932b8f6917e68f6bfedfa952b8e4acb25387607edf10856cc994021943671086af4d751cc728

data/CHANGELOG.md

@@ -1,7 +1,32 @@
+## 5.0.0 (2022-02-21)
+
+- Searches now use lazy loading (similar to Active Record)
+- Added `unscope` option to better support working with default scopes
+- Added support for `:async` and `:queue` modes for `reindex` on relation
+- Added basic protection from unfiltered parameters to `where` option
+- Added `models` option to `similar` method
+- Changed async full reindex to fetch ids instead of using ranges for numeric primary keys with Active Record
+- Changed `searchkick_index_options` to return symbol keys (instead of mix of strings and symbols)
+- Changed non-anchored regular expressions to match expected results (previously warned)
+- Changed record reindex to return `true` to match model and relation reindex
+- Updated async reindex job to call `search_import` for nested associations
+- Fixed removing records when `should_index?` is `false` when `reindex` called on relation
+- Fixed issue with `merge_mappings` for fields that use `searchkick` options
+- Raise error when `search` called on relations
+- Raise `ArgumentError` (instead of warning) for invalid regular expression modifiers
+- Raise `ArgumentError` instead of `RuntimeError` for unknown operators
+- Removed mapping of `id` to `_id` with `order` option
+- Removed `wordnet` option (no longer worked)
+- Removed dependency on `elasticsearch` gem (can use `elasticsearch` or `opensearch-ruby`)
+- Dropped support for Elasticsearch 6
+- Dropped support for Ruby < 2.6 and Active Record < 5.2
+- Dropped support for NoBrainer and Cequel
+- Dropped support for `faraday_middleware-aws-signers-v4` (use `faraday_middleware-aws-sigv4` instead)
+
 ## 4.6.3 (2021-11-19)
 
 - Added support for reloadable synonyms for OpenSearch
-- Added experimental support for `opensearch` gem
+- Added experimental support for `opensearch-ruby` gem
 - Removed `elasticsearch-xpack` dependency for reloadable synonyms
 
 ## 4.6.2 (2021-11-15)

data/README.md

@@ -20,7 +20,7 @@ Plus:
 - autocomplete
 - “Did you mean” suggestions
 - supports many languages
-- works with Active Record, Mongoid, and NoBrainer
+- works with Active Record and Mongoid
 
 Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Autosuggest](https://github.com/ankane/autosuggest) for query suggestions
 
@@ -39,26 +39,34 @@ Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Au
 - [Testing](#testing)
 - [Deployment](#deployment)
 - [Performance](#performance)
-- [Elasticsearch DSL](#advanced)
+- [Advanced Search](#advanced)
 - [Reference](#reference)
 - [Contributing](#contributing)
 
+Searchkick 5.0 was recently released! See [how to upgrade](#upgrading)
+
 ## Getting Started
 
 Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html). For Homebrew, use:
 
 ```sh
-brew install elasticsearch # or opensearch
-brew services start elasticsearch # or opensearch
+brew install elasticsearch
+brew services start elasticsearch
+# or
+brew install opensearch
+brew services start opensearch
 ```
 
-Add this line to your application’s Gemfile:
+Add these lines to your application’s Gemfile:
 
 ```ruby
-gem 'searchkick'
+gem "searchkick"
+
+gem "elasticsearch"   # select one
+gem "opensearch-ruby" # select one
 ```
 
-The latest version works with Elasticsearch 6 and 7 and OpenSearch 1. For Elasticsearch 5, use version 3.1.3 and [this readme](https://github.com/ankane/searchkick/blob/v3.1.3/README.md).
+The latest version works with Elasticsearch 7 and 8 and OpenSearch 1. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
 
 Add searchkick to models you want to search.
 
@@ -83,7 +91,7 @@ products.each do |product|
 end
 ```
 
-Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). As your search becomes more advanced, we recommend you use the [Elasticsearch DSL](#advanced) for maximum flexibility.
+Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch Search API](https://opensearch.org/docs/latest/opensearch/rest-api/search/). As your search becomes more advanced, we recommend you use the [search server DSL](#advanced) for maximum flexibility.
 
 ## Querying
 
@@ -144,7 +152,7 @@ select: [:name]
 
 ### Results
 
-Searches return a `Searchkick::Results` object. This responds like an array to most methods.
+Searches return a `Searchkick::Relation` object. This responds like an array to most methods.
 
 ```ruby
 results = Product.search("milk")
@@ -153,7 +161,7 @@ results.any?
 results.each { |result| ... }
 ```
 
-By default, ids are fetched from Elasticsearch and records are fetched from your database. To fetch everything from Elasticsearch, use:
+By default, ids are fetched from the search server and records are fetched from your database. To fetch everything from the search server, use:
 
 ```ruby
 Product.search("apples", load: false)
@@ -171,13 +179,13 @@ Get the time the search took (in milliseconds)
 results.took
 ```
 
-Get the full response from Elasticsearch
+Get the full response from the search server
 
 ```ruby
 results.response
 ```
 
-**Note:** By default, Elasticsearch [limits paging](#deep-paging) to the first 10,000 results for performance. With Elasticsearch 7, this applies to the total count as well.
+**Note:** By default, Elasticsearch and OpenSearch [limit paging](#deep-paging) to the first 10,000 results for performance. This applies to the total count as well.
 
 ### Boosting
 
@@ -375,9 +383,9 @@ search_synonyms: ["lightbulb => halogenlamp"]
 
 The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
 
-#### Elasticsearch 7.3+ or OpenSearch
+#### Elasticsearch 7.3+ and OpenSearch
 
-For Elasticsearch 7.3+ or OpenSearch, we recommend placing synonyms in a file on the Elasticsearch or OpenSearch server (in the `config` directory). This allows you to reload synonyms without reindexing.
+For Elasticsearch 7.3+ and OpenSearch, we recommend placing synonyms in a file on the search server (in the `config` directory). This allows you to reload synonyms without reindexing.
 
 ```txt
 pop, soda
@@ -485,7 +493,7 @@ Search :ice_cream::cake: and get `ice cream cake`!
 Add this line to your application’s Gemfile:
 
 ```ruby
-gem 'gemoji-parser'
+gem "gemoji-parser"
 ```
 
 And use:
@@ -520,12 +528,10 @@ class Product < ApplicationRecord
 end
 ```
 
-By default, all records are indexed. To control which records are indexed, use the `should_index?` method together with the `search_import` scope.
+By default, all records are indexed. To control which records are indexed, use the `should_index?` method.
 
 ```ruby
 class Product < ApplicationRecord
-  scope :search_import, -> { where(active: true) }
-
   def should_index?
     active # only index active records
   end
@@ -618,6 +624,28 @@ class Image < ApplicationRecord
 end
 ```
 
+### Default Scopes
+
+If you have a default scope that filters records, use the `should_index?` method to exclude them from indexing:
+
+```ruby
+class Product < ApplicationRecord
+  default_scope { where(deleted_at: nil) }
+
+  def should_index?
+    deleted_at.nil?
+  end
+end
+```
+
+If you want to index and search filtered records, set:
+
+```ruby
+class Product < ApplicationRecord
+  searchkick unscope: true
+end
+```
+
 ## Intelligent Search
 
 The best starting point to improve your search **by far** is to track searches and conversions. [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
@@ -1043,13 +1071,13 @@ Product.search("soap", debug: true)
 
 This prints useful info to `stdout`.
 
-See how Elasticsearch scores your queries with:
+See how the search server scores your queries with:
 
 ```ruby
 Product.search("soap", explain: true).response
 ```
 
-See how Elasticsearch tokenizes your queries with:
+See how the search server tokenizes your queries with:
 
 ```ruby
 Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
@@ -1223,7 +1251,7 @@ And [setup-opensearch](https://github.com/ankane/setup-opensearch) for an easy w
 
 ## Deployment
 
-Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the Elasticsearch server. This defaults to `http://localhost:9200`.
+For the search server, Searchkick uses `ENV["ELASTICSEARCH_URL"]` for Elasticsearch and `ENV["OPENSEARCH_URL"]` for OpenSearch. This defaults to `http://localhost:9200`.
 
 - [Elastic Cloud](#elastic-cloud)
 - [Heroku](#heroku)
@@ -1248,13 +1276,20 @@ rake searchkick:reindex:all
 
 Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).
 
-For Bonsai:
+For Elasticsearch on Bonsai:
 
 ```sh
-heroku addons:create bonsai # use --engine=opensearch for OpenSearch
+heroku addons:create bonsai
 heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
 ```
 
+For OpenSearch on Bonsai:
+
+```sh
+heroku addons:create bonsai --engine=opensearch
+heroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`
+```
+
 For SearchBox:
 
 ```sh
@@ -1289,16 +1324,16 @@ heroku run rake searchkick:reindex:all
 
 ### Amazon OpenSearch Service
 
-Create an initializer `config/initializers/elasticsearch.rb` with:
+Create an initializer `config/initializers/opensearch.rb` with:
 
 ```ruby
-ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
+ENV["OPENSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
 ```
 
 To use signed requests, include in your Gemfile:
 
 ```ruby
-gem 'faraday_middleware-aws-sigv4'
+gem "faraday_middleware-aws-sigv4"
 ```
 
 and add to your initializer:
@@ -1319,10 +1354,12 @@ rake searchkick:reindex:all
 
 ### Self-Hosted and Other
 
-Create an initializer `config/initializers/elasticsearch.rb` with:
+Create an initializer with:
 
 ```ruby
 ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
+# or
+ENV["OPENSEARCH_URL"] = "https://user:password@host:port"
 ```
 
 Then deploy and reindex:
@@ -1333,9 +1370,9 @@ rake searchkick:reindex:all
 
 ### Data Protection
 
-We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to Elasticsearch.
+We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to the search server.
 
-Bonsai, Elastic Cloud, and Amazon Elasticsearch all support encryption at rest and HTTPS.
+Bonsai, Elastic Cloud, and Amazon OpenSearch Service all support encryption at rest and HTTPS.
 
 ### Automatic Failover
 
@@ -1368,7 +1405,7 @@ See [Production Rails](https://github.com/ankane/production_rails) for other goo
 Significantly increase performance with faster JSON generation. Add [Oj](https://github.com/ohler55/oj) to your Gemfile.
 
 ```ruby
-gem 'oj'
+gem "oj"
 ```
 
 This speeds up all JSON generation and parsing in your application (automatically!)
@@ -1378,7 +1415,7 @@ This speeds up all JSON generation and parsing in your application (automaticall
 Significantly increase performance with persistent HTTP connections. Add [Typhoeus](https://github.com/typhoeus/typhoeus) to your Gemfile and it’ll automatically be used.
 
 ```ruby
-gem 'typhoeus'
+gem "typhoeus"
 ```
 
 To reduce log noise, create an initializer with:
@@ -1447,7 +1484,7 @@ Product.reindex(async: {wait: true})
 You can use [ActiveJob::TrafficControl](https://github.com/nickelser/activejob-traffic_control) to control concurrency. Install the gem:
 
 ```ruby
-gem 'activejob-traffic_control', '>= 0.1.3'
+gem "activejob-traffic_control", ">= 0.1.3"
 ```
 
 And create an initializer with:
@@ -1510,7 +1547,7 @@ For more tips, check out [Keeping Elasticsearch in Sync](https://www.elastic.co/
 
 ### Routing
 
-Searchkick supports [Elasticsearch’s routing feature](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
+Searchkick supports [routing](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
 
 ```ruby
 class Business < ApplicationRecord
@@ -1618,7 +1655,7 @@ ReindexConversionsJob.perform_later("Product")
 
 ## Advanced
 
-Searchkick makes it easy to use the Elasticsearch DSL on its own.
+Searchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.
 
 ### Advanced Mapping
 
@@ -1672,7 +1709,7 @@ products =
   end
 ```
 
-### Elasticsearch Gem
+### Client
 
 Searchkick is built on top of the [elasticsearch](https://github.com/elastic/elasticsearch-ruby) gem. To access the client directly, use:
 
@@ -1685,8 +1722,8 @@ Searchkick.client
 To batch search requests for performance, use:
 
 ```ruby
-products = Product.search("snacks", execute: false)
-coupons = Coupon.search("snacks", execute: false)
+products = Product.search("snacks")
+coupons = Coupon.search("snacks")
 Searchkick.multi_search([products, coupons])
 ```
 
@@ -1737,7 +1774,7 @@ products.clear_scroll
 
 ## Deep Paging
 
-By default, Elasticsearch limits paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:
+By default, Elasticsearch and OpenSearch limit paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:
 
 ```ruby
 class Product < ApplicationRecord
@@ -1745,7 +1782,7 @@ class Product < ApplicationRecord
 end
 ```
 
-If you just need an accurate total count with Elasticsearch 7, you can instead use:
+If you just need an accurate total count, you can instead use:
 
 ```ruby
 Product.search("pears", body_options: {track_total_hits: true})
@@ -1960,13 +1997,6 @@ class Product < ApplicationRecord
 end
 ```
 
-Lazy searching
-
-```ruby
-products = Product.search("carrots", execute: false)
-products.each { ... } # search not executed until here
-```
-
 Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-search-api-query-params) like `search_type`
 
 ```ruby
@@ -2004,22 +2034,17 @@ Turn on misspellings after a certain number of characters
 Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi
 ```
 
-**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off
+**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch
 
 ```ruby
 Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
 ```
 
-## Elasticsearch 6 to 7 Upgrade
-
-1. Install Searchkick 4
-2. Upgrade your Elasticsearch cluster
-
-## Elasticsearch Gotchas
+## Gotchas
 
 ### Consistency
 
-Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
+Elasticsearch and OpenSearch are eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
 
 ```ruby
 product.save!
@@ -2028,7 +2053,7 @@ Product.search_index.refresh
 
 ### Inconsistent Scores
 
-Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
+Due to the distributed nature of Elasticsearch and OpenSearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
 
 ```ruby
 class Product < ApplicationRecord
@@ -2038,6 +2063,34 @@ end
 
 For convenience, this is set by default in the test environment.
 
+## Upgrading
+
+### 5.0
+
+Searchkick 5 supports both the `elasticsearch` and `opensearch-ruby` gems. Add the one you want to use to your Gemfile:
+
+```ruby
+gem "elasticsearch"
+# or
+gem "opensearch-ruby"
+```
+
+If using the deprecated `faraday_middleware-aws-signers-v4` gem, switch to `faraday_middleware-aws-sigv4`.
+
+Also, searches now use lazy loading:
+
+```ruby
+# search not executed
+Product.search("milk")
+
+# search executed
+Product.search("milk").to_a
+```
+
+And there’s a [new option](#default-scopes) for models with default scopes.
+
+Check out the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md) for the full list of changes.
+
 ## History
 
 View the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md).

data/lib/searchkick/bulk_reindex_job.rb

@@ -2,16 +2,20 @@ module Searchkick
   class BulkReindexJob < ActiveJob::Base
     queue_as { Searchkick.queue_name }
 
+    # TODO remove min_id and max_id in Searchkick 6
     def perform(class_name:, record_ids: nil, index_name: nil, method_name: nil, batch_id: nil, min_id: nil, max_id: nil)
-      klass = class_name.constantize
-      index = index_name ? Searchkick::Index.new(index_name, **klass.searchkick_options) : klass.searchkick_index
+      model = Searchkick.load_model(class_name)
+      index = model.searchkick_index(name: index_name)
+
+      # legacy
       record_ids ||= min_id..max_id
-      index.import_scope(
-        Searchkick.load_records(klass, record_ids),
-        method_name: method_name,
-        batch: true,
-        batch_id: batch_id
-      )
+
+      relation = Searchkick.scope(model)
+      relation = Searchkick.load_records(relation, record_ids)
+      relation = relation.search_import if relation.respond_to?(:search_import)
+
+      RecordIndexer.new(index).reindex(relation, mode: :inline, method_name: method_name, full: false)
+      RelationIndexer.new(index).batch_completed(batch_id) if batch_id
     end
   end
 end

data/lib/searchkick/controller_runtime.rb

@@ -0,0 +1,40 @@
+# based on https://gist.github.com/mnutt/566725
+module Searchkick
+  module ControllerRuntime
+    extend ActiveSupport::Concern
+
+    protected
+
+    attr_internal :searchkick_runtime
+
+    def process_action(action, *args)
+      # We also need to reset the runtime before each action
+      # because of queries in middleware or in cases we are streaming
+      # and it won't be cleaned up by the method below.
+      Searchkick::LogSubscriber.reset_runtime
+      super
+    end
+
+    def cleanup_view_runtime
+      searchkick_rt_before_render = Searchkick::LogSubscriber.reset_runtime
+      runtime = super
+      searchkick_rt_after_render = Searchkick::LogSubscriber.reset_runtime
+      self.searchkick_runtime = searchkick_rt_before_render + searchkick_rt_after_render
+      runtime - searchkick_rt_after_render
+    end
+
+    def append_info_to_payload(payload)
+      super
+      payload[:searchkick_runtime] = (searchkick_runtime || 0) + Searchkick::LogSubscriber.reset_runtime
+    end
+
+    module ClassMethods
+      def log_process_action(payload)
+        messages = super
+        runtime = payload[:searchkick_runtime]
+        messages << ("Searchkick: %.1fms" % runtime.to_f) if runtime.to_f > 0
+        messages
+      end
+    end
+  end
+end