chewy 7.2.1 → 7.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70c6d061a7070669eafe4f28bd09cd8e7a9138d0a66dd6fdbdf1f387f3f3f0af
-  data.tar.gz: '03838f9c5e5b323e68482066b057bccff969e2826e581b994b92a962bb45e272'
+  metadata.gz: 229c412b1e3464060b367c9c378d3662bccf2329b4e4eaa7617541243dbd6e38
+  data.tar.gz: 9bc674550cd6152e8a18853fa46c91873c7a626c426cd50aa2cb8c67b1c9455e
 SHA512:
-  metadata.gz: 8fcafb7b237abf426ed941e6b8d53b537db65186ce98f40b6c1b50257bc570b500b9bbff71aedb6828dfb0be11e13f031f3cf92413fd0f3efb5b860deca53a28
-  data.tar.gz: c74fe5e6db07346b9e77f4a0d294317a6ade800fa506f231d7368058a6f058785bb0c9756110378b810398a7bc24ca5c695bbb46c330a97976fd3b1bf6a8d3da
+  metadata.gz: e3e8409f81bc2504e9edbf5093fc9329bf443862cb96e113c9c00fc586c1e0e3a4273b3d6e82466f9b3d301e3d9eafcc72892cce718aff7dcfbbce85aba1698d
+  data.tar.gz: 33b5b0e21ef7c9896f1ed381130f225a1d51e073112213f631fb1c57c6d69c49af54817b1de3afc9be325c79ac24f9c173275b4448ffb7b43e82403bee03bd70

data/CHANGELOG.md

@@ -8,6 +8,25 @@
 
 ### Bugs Fixed
 
+## 7.2.2 (2021-05-24)
+
+### Changes
+
+  * [#800](https://github.com/toptal/chewy/pull/800): Revert [#787](https://github.com/toptal/chewy/pull/787) progressbar feature to avoid performance degradation in parallel import ([@rabotyaga][])
+
+  * [#795](https://github.com/toptal/chewy/issues/795): **(Breaking)** Change the Chewy::Search::Parameters::Order implementation to use Array ([@jiajiawang][]):
+    * To allow multiple sorting options that may have the same key name. For example script based sorting whose key will always be `_script`.
+    * Behaviour change of chained `order` calls.
+      * e.g. `.order(_script: {a: 1}).order(_script: {b: 2})`
+        * Before `{:sort=>[{"_script"=>{:b=>2}}]}`
+        * After  `{:sort=>[{"_script"=>{:a=>1}},{"_script"=>{:b=>2}}]}`
+
+  * [#654](https://github.com/toptal/chewy/issues/654): Add helpers and matchers for testing ([@Vitalina-Vakulchyk][]):
+    * `mock_elasticsearch_response` helpers both Rspec and Minitest - to mock elasticsearch response
+    * `mock_elasticsearch_response_sources` helpers both Rspec and Minitest - to mock elasticsearch response sources
+    * `assert_elasticsearch_query` helper for Minitest - to compare request and expected query (returns `true`/`false`)
+    * `build_query` matcher for Rspec - to compare request and expected query (returns `true`/`false`)
+
 ## 7.2.1 (2021-05-11)
 
 ### New Features
@@ -631,6 +650,7 @@
 [@inbeom]: https://github.com/inbeom
 [@jesjos]: https://github.com/jesjos
 [@JF-Lalonde]: https://github.com/JF-Lalonde
+[@jiajiawang]: https://github.com/jiajiawang
 [@jimmybaker]: https://github.com/jimmybaker
 [@jirikolarik]: https://github.com/jirikolarik
 [@jirutka]: https://github.com/jirutka

data/README.md

@@ -53,7 +53,7 @@ Chewy is compatible with MRI 2.6-3.0¹.
 | ------------- | ---------------------------------- |
 | 7.2.x         | 7.x                                |
 | 7.1.x         | 7.x                                |
-| 7.0.0         | 6.8, 7.x                           |
+| 7.0.x         | 6.8, 7.x                           |
 | 6.0.0         | 5.x, 6.x                           |
 | 5.x           | 5.x, limited support for 1.x & 2.x |
 
@@ -1032,12 +1032,6 @@ rake chewy:reset[users,cities] # resets UsersIndex and CitiesIndex
 rake chewy:reset[-users,cities] # resets every index in the application except specified ones
 ```
 
-#### Progressbar for `chewy:reset` tasks
-
-You can optionally output the `progressbar` for `chewy:reset` and `chewy:parallel:reset` during import.
-
-Progressbar is hidden by default. Set `ENV['PROGRESS']` to `true` to display it.
-
 #### `chewy:upgrade`
 
 Performs reset exactly the same way as `chewy:reset` does, but only when the index specification (setting or mapping) was changed.
@@ -1117,7 +1111,16 @@ rake chewy:journal:apply["$(date -v-1H -u +%FT%TZ)",users] # apply journaled cha
 
 ### RSpec integration
 
-Just add `require 'chewy/rspec'` to your spec_helper.rb and you will get additional features: See [update_index.rb](lib/chewy/rspec/update_index.rb) for more details.
+Just add `require 'chewy/rspec'` to your spec_helper.rb and you will get additional features:
+
+[update_index](lib/chewy/rspec/update_index.rb) helper
+`mock_elasticsearch_response` helper to mock elasticsearch response
+`mock_elasticsearch_response_sources` helper to mock elasticsearch response sources
+`build_query` matcher to compare request and expected query (returns `true`/`false`)
+
+To use `mock_elasticsearch_response` and `mock_elasticsearch_response_sources` helpers add `include Chewy::Rspec::Helpers` to your tests.
+
+See [chewy/rspec/](lib/chewy/rspec/) for more details.
 
 ### Minitest integration
 
@@ -1127,6 +1130,14 @@ Since you can set `:bypass` strategy for test suites and manually handle import
 
 But if you require chewy to index/update model regularly in your test suite then you can specify `:urgent` strategy for documents indexing. Add `Chewy.strategy(:urgent)` to test_helper.rb.
 
+Also, you can use additional helpers:
+
+`mock_elasticsearch_response` to mock elasticsearch response
+`mock_elasticsearch_response_sources` to mock elasticsearch response sources
+`assert_elasticsearch_query` to compare request and expected query (returns `true`/`false`)
+
+See [chewy/minitest/](lib/chewy/minitest/) for more details.
+
 ### DatabaseCleaner
 
 If you use `DatabaseCleaner` in your tests with [the `transaction` strategy](https://github.com/DatabaseCleaner/database_cleaner#how-to-use), you may run into the problem that `ActiveRecord`'s models are not indexed automatically on save despite the fact that you set the callbacks to do this with the `update_index` method. The issue arises because `chewy` indices data on `after_commit` run as default, but all `after_commit` callbacks are not run with the `DatabaseCleaner`'s' `transaction` strategy. You can solve this issue by changing the `Chewy.use_after_commit_callbacks` option. Just add the following initializer in your Rails application:

data/chewy.gemspec

@@ -31,7 +31,6 @@ Gem::Specification.new do |spec| # rubocop:disable Metrics/BlockLength
   spec.add_development_dependency 'unparser'
 
   spec.add_dependency 'activesupport', '>= 5.2'
-  spec.add_dependency 'elasticsearch', '>= 6.3.0'
+  spec.add_dependency 'elasticsearch', '>= 7.12.0'
   spec.add_dependency 'elasticsearch-dsl'
-  spec.add_dependency 'ruby-progressbar'
 end

data/lib/chewy/index/actions.rb

@@ -146,7 +146,7 @@ module Chewy
         # @param journal [true, false] journaling is switched off for import during reset by default
         # @param import_options [Hash] options, passed to the import call
         # @return [true, false] false in case of errors
-        def reset!(suffix = nil, apply_journal: true, journal: false, progressbar: false, **import_options)
+        def reset!(suffix = nil, apply_journal: true, journal: false, **import_options)
           result = if suffix.present?
             start_time = Time.now
             indexes = self.indexes - [index_name]
@@ -159,13 +159,17 @@ module Chewy
             result = import import_options.merge(
               suffix: suffix,
               journal: journal,
-              refresh: !Chewy.reset_disable_refresh_interval,
-              progressbar: progressbar
+              refresh: !Chewy.reset_disable_refresh_interval
             )
             original_index_settings suffixed_name
 
             delete if indexes.blank?
-            update_aliases(indexes, general_name, suffixed_name)
+            client.indices.update_aliases body: {actions: [
+              *indexes.map do |index|
+                {remove: {index: index, alias: general_name}}
+              end,
+              {add: {index: suffixed_name, alias: general_name}}
+            ]}
             client.indices.delete index: indexes if indexes.present?
 
             self.journal.apply(start_time, **import_options) if apply_journal
@@ -233,15 +237,6 @@ module Chewy
 
       private
 
-        def update_aliases(indexes, general_name, suffixed_name)
-          client.indices.update_aliases body: {actions: [
-            *indexes.map do |index|
-              {remove: {index: index, alias: general_name}}
-            end,
-            {add: {index: suffixed_name, alias: general_name}}
-          ]}
-        end
-
         def optimize_index_settings(index_name)
           settings = {}
           settings[:refresh_interval] = -1 if Chewy.reset_disable_refresh_interval

data/lib/chewy/index/adapter/object.rb

@@ -192,16 +192,6 @@ module Chewy
           end
         end
 
-        def import_count(*args)
-          collection = if args.first.empty? && @target.respond_to?(import_all_method)
-            @target.send(import_all_method)
-          else
-            args.flatten(1).compact
-          end
-
-          collection.count
-        end
-
       private
 
         def import_objects(objects, options)

data/lib/chewy/index/adapter/orm.rb

@@ -108,18 +108,6 @@ module Chewy
           ids.map { |id| loaded_objects[id.to_s] }
         end
 
-        def import_count(*args)
-          collection = if args.first.empty?
-            default_scope
-          elsif args.first.is_a?(relation_class)
-            args.first
-          else
-            args.flatten.compact
-          end
-
-          collection.count
-        end
-
       private
 
         def import_objects(collection, options)

data/lib/chewy/index/import.rb

@@ -2,19 +2,17 @@ require 'chewy/index/import/journal_builder'
 require 'chewy/index/import/bulk_builder'
 require 'chewy/index/import/bulk_request'
 require 'chewy/index/import/routine'
-require 'chewy/index/import/thread_safe_progress_bar'
 
 module Chewy
   class Index
     module Import
       extend ActiveSupport::Concern
 
-      IMPORT_WORKER = lambda do |index, options, total, progress_bar, ids, iteration|
+      IMPORT_WORKER = lambda do |index, options, total, ids, iteration|
         ::Process.setproctitle("chewy [#{index}]: import data (#{iteration + 1}/#{total})")
         routine = Routine.new(index, **options)
         index.adapter.import(*ids, routine.options) do |action_objects|
           routine.process(**action_objects)
-          progress_bar.increment(action_objects.map { |_, v| v.size }.sum) if routine.options[:progressbar]
         end
         {errors: routine.errors, import: routine.stats, leftovers: routine.leftovers}
       end
@@ -152,10 +150,8 @@ module Chewy
 
         def import_linear(objects, routine)
           ActiveSupport::Notifications.instrument 'import_objects.chewy', index: self do |payload|
-            progress_bar = ThreadSafeProgressBar.new(routine.options[:progressbar]) { adapter.import_count(objects) }
             adapter.import(*objects, routine.options) do |action_objects|
               routine.process(**action_objects)
-              progress_bar.increment(action_objects.map { |_, v| v.size }.sum) if routine.options[:progressbar]
             end
             routine.perform_bulk(routine.leftovers)
             payload[:import] = routine.stats
@@ -170,20 +166,24 @@ module Chewy
           ActiveSupport::Notifications.instrument 'import_objects.chewy', index: self do |payload|
             batches = adapter.import_references(*objects, routine.options.slice(:batch_size)).to_a
 
-            progress_bar = ThreadSafeProgressBar.new(routine.options[:progressbar]) { adapter.import_count(objects) }
             ::ActiveRecord::Base.connection.close if defined?(::ActiveRecord::Base)
-
-            results = ::Parallel.map_with_index(batches, routine.parallel_options) do |ids, index|
-              progress_bar.wait_until_ready
-              ActiveRecord::Base.connection_pool.with_connection do
-                IMPORT_WORKER.call(self, routine.options, total, progress_bar, ids, index)
-              end
-            end
-
+            results = ::Parallel.map_with_index(
+              batches,
+              routine.parallel_options,
+              &IMPORT_WORKER.curry[self, routine.options, batches.size]
+            )
             ::ActiveRecord::Base.connection.reconnect! if defined?(::ActiveRecord::Base)
             errors, import, leftovers = process_parallel_import_results(results)
 
-            execute_leftovers(leftovers, routine, self, errors)
+            if leftovers.present?
+              batches = leftovers.each_slice(routine.options[:batch_size])
+              results = ::Parallel.map_with_index(
+                batches,
+                routine.parallel_options,
+                &LEFTOVERS_WORKER.curry[self, routine.options, batches.size]
+              )
+              errors.concat(results.flatten(1))
+            end
 
             payload[:import] = import
             payload[:errors] = payload_errors(errors) if errors.present?
@@ -191,18 +191,6 @@ module Chewy
           end
         end
 
-        def execute_leftovers(leftovers, routine, self_object, errors)
-          return unless leftovers.present?
-
-          batches = leftovers.each_slice(routine.options[:batch_size])
-          results = ::Parallel.map_with_index(
-            batches,
-            routine.parallel_options,
-            &LEFTOVERS_WORKER.curry[self_object, routine.options, batches.size]
-          )
-          errors.concat(results.flatten(1))
-        end
-
         def process_parallel_import_results(results)
           results.each_with_object([[], {}, []]) do |r, (e, i, l)|
             e.concat(r[:errors])

data/lib/chewy/index/import/routine.rb

@@ -51,9 +51,9 @@ module Chewy
           @parallel_options = @options.delete(:parallel)
           if @parallel_options && !@parallel_options.is_a?(Hash)
             @parallel_options = if @parallel_options.is_a?(Integer)
-              {in_threads: @parallel_options}
+              {in_processes: @parallel_options}
             else
-              {in_threads: [::Parallel.processor_count, ActiveRecord::Base.connection_pool.size].min}
+              {}
             end
           end
           @errors = []

data/lib/chewy/minitest/helpers.rb

@@ -42,10 +42,87 @@ module Chewy
       # Run indexing for the database changes during the block provided.
       # By default, indexing is run at the end of the block.
       # @param strategy [Symbol] the Chewy index update strategy see Chewy docs.
+      #
       def run_indexing(strategy: :atomic, &block)
         Chewy.strategy strategy, &block
       end
 
+      # Mock Elasticsearch response
+      # Simple usage - just pass index, expected raw response
+      # and block with the query.
+      #
+      # @param index [Chewy::Index] the index to watch, eg EntitiesIndex.
+      # @param raw_response [Hash] hash with response.
+      #
+      def mock_elasticsearch_response(index, raw_response)
+        mocked_request = Chewy::Search::Request.new(index)
+
+        original_new = Chewy::Search::Request.method(:new)
+
+        Chewy::Search::Request.define_singleton_method(:new) { |*_args| mocked_request }
+
+        original_perform = mocked_request.method(:perform)
+        mocked_request.define_singleton_method(:perform) { raw_response }
+
+        yield
+      ensure
+        mocked_request.define_singleton_method(:perform, original_perform)
+        Chewy::Search::Request.define_singleton_method(:new, original_new)
+      end
+
+      # Mock Elasticsearch response with defined sources
+      # Simple usage - just pass index, expected sources
+      # and block with the query.
+      #
+      # @param index [Chewy::Index] the index to watch, eg EntitiesIndex.
+      # @param hits [Hash] hash with sources.
+      #
+      def mock_elasticsearch_response_sources(index, hits, &block)
+        raw_response = {
+          'took' => 4,
+          'timed_out' => false,
+          '_shards' => {
+            'total' => 1,
+            'successful' => 1,
+            'skipped' => 0,
+            'failed' => 0
+          },
+          'hits' => {
+            'total' => {
+              'value' => hits.count,
+              'relation' => 'eq'
+            },
+            'max_score' => 1.0,
+            'hits' => hits.each_with_index.map do |hit, i|
+              {
+                '_index' => index.index_name,
+                '_type' => '_doc',
+                '_id' => (i + 1).to_s,
+                '_score' => 3.14,
+                '_source' => hit
+              }
+            end
+          }
+        }
+
+        mock_elasticsearch_response(index, raw_response, &block)
+      end
+
+      # Check the assertion that actual Elasticsearch query is rendered
+      # to the expected query
+      #
+      # @param query [::Query] the actual Elasticsearch query.
+      # @param expected_query [Hash] expected query.
+      #
+      # @return [Boolean]
+      #   True - in the case when actual Elasticsearch query is rendered to the expected query.
+      #   False - in the opposite case.
+      #
+      def assert_elasticsearch_query(query, expected_query)
+        actual_query = query.render
+        assert_equal expected_query, actual_query, "got #{actual_query.inspect} instead of expected query."
+      end
+
       module ClassMethods
         # Declare that all tests in this file require real indexing, always.
         # In my completely unscientific experiments, this roughly doubled test runtime.

data/lib/chewy/rake_helper.rb

@@ -271,7 +271,7 @@ module Chewy
 
       def reset_one(index, output, parallel: false)
         output.puts "Resetting #{index}"
-        index.reset!((Time.now.to_f * 1000).round, parallel: parallel, progressbar: ENV['PROGRESS'] == 'true')
+        index.reset!((Time.now.to_f * 1000).round, parallel: parallel)
       end
     end
   end

data/lib/chewy/rspec.rb

@@ -1 +1,3 @@
+require 'chewy/rspec/build_query'
+require 'chewy/rspec/helpers'
 require 'chewy/rspec/update_index'

data/lib/chewy/rspec/build_query.rb

@@ -0,0 +1,12 @@
+# Rspec helper to compare request and expected query
+# To use it - add `require 'chewy/rspec/build_query'` to the `spec_helper.rb`
+# Simple usage - just pass expected response as argument
+# and then call needed query.
+#
+#   expect { method1.method2...methodN }.to build_query(expected_query)
+#
+RSpec::Matchers.define :build_query do |expected_query = {}|
+  match do |request|
+    request.render == expected_query
+  end
+end

data/lib/chewy/rspec/helpers.rb

@@ -0,0 +1,55 @@
+module Chewy
+  module Rspec
+    module Helpers
+      extend ActiveSupport::Concern
+      # Rspec helper to mock elasticsearch response
+      # To use it - add `require 'chewy/rspec'` to the `spec_helper.rb`
+      #
+      #   mock_elasticsearch_response(CitiesIndex, raw_response)
+      #   expect(CitiesIndex.query({}).hits).to eq(hits)
+      #
+      def mock_elasticsearch_response(index, raw_response)
+        mocked_request = Chewy::Search::Request.new(index)
+        allow(Chewy::Search::Request).to receive(:new).and_return(mocked_request)
+        allow(mocked_request).to receive(:perform).and_return(raw_response)
+      end
+
+      # Rspec helper to mock Elasticsearch response source
+      # To use it - add `require 'chewy/rspec'` to the `spec_helper.rb`
+      #
+      #  mock_elasticsearch_response_sources(CitiesIndex, sources)
+      #  expect(CitiesIndex.query({}).hits).to eq(hits)
+      #
+      def mock_elasticsearch_response_sources(index, hits)
+        raw_response = {
+          'took' => 4,
+          'timed_out' => false,
+          '_shards' => {
+            'total' => 1,
+            'successful' => 1,
+            'skipped' => 0,
+            'failed' => 0
+          },
+          'hits' => {
+            'total' => {
+              'value' => hits.count,
+              'relation' => 'eq'
+            },
+            'max_score' => 1.0,
+            'hits' => hits.each_with_index.map do |hit, i|
+              {
+                '_index' => index.index_name,
+                '_type' => '_doc',
+                '_id' => (i + 1).to_s,
+                '_score' => 3.14,
+                '_source' => hit
+              }
+            end
+          }
+        }
+
+        mock_elasticsearch_response(index, raw_response)
+      end
+    end
+  end
+end

data/lib/chewy/search/parameters/order.rb

@@ -17,7 +17,7 @@ module Chewy
         # @param other_value [Object] any acceptable storage value
         # @return [Object] updated value
         def update!(other_value)
-          value.merge!(normalize(other_value))
+          value.concat(normalize(other_value))
         end
 
         # Size requires specialized rendering logic, it should return
@@ -28,20 +28,7 @@ module Chewy
         def render
           return if value.blank?
 
-          sort = value.map do |(field, options)|
-            options ? {field => options} : field
-          end
-          {sort: sort}
-        end
-
-        # Comparison also reqires additional logic. Hashes are compared
-        # orderlessly, but for `sort` parameter oder is important, so we
-        # compare hash key collections additionally.
-        #
-        # @see Chewy::Search::Parameters::Storage#==
-        # @return [true, false]
-        def ==(other)
-          super && value.keys == other.value.keys
+          {sort: value}
         end
 
       private
@@ -49,13 +36,13 @@ module Chewy
         def normalize(value)
           case value
           when Array
-            value.each_with_object({}) do |sv, res|
-              res.merge!(normalize(sv))
+            value.each_with_object([]) do |sv, res|
+              res.concat(normalize(sv))
             end
           when Hash
-            value.stringify_keys
+            [value.stringify_keys]
           else
-            value.present? ? {value.to_s => nil} : {}
+            value.present? ? [value.to_s] : []
           end
         end
       end

data/lib/chewy/version.rb

@@ -1,3 +1,3 @@
 module Chewy
-  VERSION = '7.2.1'.freeze
+  VERSION = '7.2.2'.freeze
 end

data/migration_guide.md

@@ -9,7 +9,7 @@ Chewy alongside a matching Elasticsearch version.
 In order to upgrade Chewy 6/Elasticsearch 6 to Chewy 7/Elasticsearch 7 in the most seamless manner you have to:
 
 * Upgrade to the latest 6.x stable releases, namely Chewy 6.0, Elasticsearch 6.8
-* Study carefully [Breaking changes in 7.0](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.htmll), make sure your application conforms.
+* Study carefully [Breaking changes in 7.0](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html), make sure your application conforms.
 * Run your test suite on Chewy 7.0 / Elasticsearch 7
 * Run manual tests on Chewy 7.0 / Elasticsearch 7
 * Upgrade to Chewy 7.0

data/spec/chewy/index/actions_spec.rb

@@ -502,7 +502,7 @@ describe Chewy::Index::Actions do
         specify do
           expect(CitiesIndex.client.indices).to receive(:put_settings).with(index: name, body: before_import_body).once
           expect(CitiesIndex.client.indices).to receive(:put_settings).with(index: name, body: after_import_body).once
-          expect(CitiesIndex).to receive(:import).with(suffix: suffix, progressbar: false, journal: false, refresh: false).and_call_original
+          expect(CitiesIndex).to receive(:import).with(suffix: suffix, journal: false, refresh: false).and_call_original
           expect(CitiesIndex.reset!(suffix)).to eq(true)
         end
 
@@ -525,7 +525,7 @@ describe Chewy::Index::Actions do
               .to receive(:put_settings).with(index: name, body: before_import_body).once
             expect(CitiesIndex.client.indices).to receive(:put_settings).with(index: name, body: after_import_body).once
             expect(CitiesIndex)
-              .to receive(:import).with(suffix: suffix, progressbar: false, journal: false, refresh: false).and_call_original
+              .to receive(:import).with(suffix: suffix, journal: false, refresh: false).and_call_original
             expect(CitiesIndex.reset!(suffix)).to eq(true)
           end
 
@@ -541,7 +541,7 @@ describe Chewy::Index::Actions do
         let(:reset_disable_refresh_interval) { false }
         specify do
           expect(CitiesIndex.client.indices).not_to receive(:put_settings)
-          expect(CitiesIndex).to receive(:import).with(suffix: suffix, progressbar: false, journal: false, refresh: true).and_call_original
+          expect(CitiesIndex).to receive(:import).with(suffix: suffix, journal: false, refresh: true).and_call_original
           expect(CitiesIndex.reset!(suffix)).to eq(true)
         end
       end
@@ -568,7 +568,7 @@ describe Chewy::Index::Actions do
         specify do
           expect(CitiesIndex.client.indices).to receive(:put_settings).with(index: name, body: before_import_body).once
           expect(CitiesIndex.client.indices).to receive(:put_settings).with(index: name, body: after_import_body).once
-          expect(CitiesIndex).to receive(:import).with(suffix: suffix, progressbar: false, journal: false, refresh: true).and_call_original
+          expect(CitiesIndex).to receive(:import).with(suffix: suffix, journal: false, refresh: true).and_call_original
           expect(CitiesIndex.reset!(suffix)).to eq(true)
         end
       end
@@ -577,7 +577,7 @@ describe Chewy::Index::Actions do
         let(:reset_no_replicas) { false }
         specify do
           expect(CitiesIndex.client.indices).not_to receive(:put_settings)
-          expect(CitiesIndex).to receive(:import).with(suffix: suffix, progressbar: false, journal: false, refresh: true).and_call_original
+          expect(CitiesIndex).to receive(:import).with(suffix: suffix, journal: false, refresh: true).and_call_original
           expect(CitiesIndex.reset!(suffix)).to eq(true)
         end
       end
@@ -667,7 +667,7 @@ describe Chewy::Index::Actions do
       specify do
         expect(CitiesIndex)
           .to receive(:import)
-          .with(suffix: 'suffix', progressbar: false, parallel: true, journal: false, refresh: true)
+          .with(suffix: 'suffix', parallel: true, journal: false, refresh: true)
           .once.and_return(true)
         expect(CitiesIndex.reset!('suffix', parallel: true)).to eq(true)
       end

data/spec/chewy/index/import/routine_spec.rb

@@ -64,8 +64,8 @@ describe Chewy::Index::Import::Routine do
 
   describe '#parallel_options' do
     specify { expect(described_class.new(CitiesIndex).parallel_options).to be_nil }
-    specify { expect(described_class.new(CitiesIndex, parallel: true).parallel_options).to eq({in_threads: [::Parallel.processor_count, ActiveRecord::Base.connection_pool.size].min}) }
-    specify { expect(described_class.new(CitiesIndex, parallel: 3).parallel_options).to eq(in_threads: 3) }
+    specify { expect(described_class.new(CitiesIndex, parallel: true).parallel_options).to eq({}) }
+    specify { expect(described_class.new(CitiesIndex, parallel: 3).parallel_options).to eq(in_processes: 3) }
     specify do
       expect(described_class.new(CitiesIndex, parallel: {in_threads: 2}).parallel_options).to eq(in_threads: 2)
     end

data/spec/chewy/index/import_spec.rb

@@ -493,35 +493,6 @@ describe Chewy::Index::Import do
 
       it_behaves_like 'importing'
     end
-
-    context 'with progressbar output' do
-      let(:mocked_progressbar) { Struct.new(:progress, :total).new(0, 100) }
-
-      it 'imports tracks progress in a single batch' do
-        expect(ProgressBar).to receive(:create).and_return(mocked_progressbar)
-        expect(mocked_progressbar).to receive(:progress).at_least(:once).and_call_original
-        expect(CitiesIndex).to receive(:import_parallel).and_call_original
-
-        CitiesIndex.import(parallel: 1, progressbar: true)
-
-        expect(mocked_progressbar.progress).to eq(3)
-        expect(mocked_progressbar.total).to eq(3)
-      end
-
-      it 'imports tracks progress in many batches' do
-        expect(ProgressBar).to receive(:create).and_return(mocked_progressbar)
-        expect(mocked_progressbar).to receive(:progress).at_least(:once).and_call_original
-        expect(CitiesIndex).to receive(:import_parallel).and_call_original
-
-        batches = City.pluck(:id).map { |id| [id] }
-        expect(CitiesIndex.adapter).to receive(:import_references).and_return(batches)
-
-        CitiesIndex.import(parallel: 3, progressbar: true)
-
-        expect(mocked_progressbar.progress).to eq(3)
-        expect(mocked_progressbar.total).to eq(3)
-      end
-    end
   end
 
   describe '.import!', :orm do
@@ -535,9 +506,7 @@ describe Chewy::Index::Import do
         end
       end
 
-      specify do
-        expect { CitiesIndex.import!(dummy_cities) }.to raise_error Chewy::ImportFailed
-      end
+      specify { expect { CitiesIndex.import!(dummy_cities) }.to raise_error Chewy::ImportFailed }
     end
   end
 

data/spec/chewy/minitest/helpers_spec.rb

@@ -12,6 +12,10 @@ describe :minitest_helper do
 
   include ::Chewy::Minitest::Helpers
 
+  def assert_equal(expected, actual, message)
+    raise message unless expected == actual
+  end
+
   before do
     Chewy.massacre
   end
@@ -22,6 +26,112 @@ describe :minitest_helper do
     end
   end
 
+  describe 'mock_elasticsearch_response' do
+    let(:hits) do
+      [
+        {
+          '_index' => 'dummies',
+          '_type' => '_doc',
+          '_id' => '1',
+          '_score' => 3.14,
+          '_source' => source
+        }
+      ]
+    end
+
+    let(:source) { {'name' => 'some_name'} }
+    let(:sources) { [source] }
+
+    context 'mocks by raw response' do
+      let(:raw_response) do
+        {
+          'took' => 4,
+          'timed_out' => false,
+          '_shards' => {
+            'total' => 1,
+            'successful' => 1,
+            'skipped' => 0,
+            'failed' => 0
+          },
+          'hits' => {
+            'total' => {
+              'value' => 1,
+              'relation' => 'eq'
+            },
+            'max_score' => 1.0,
+            'hits' => hits
+          }
+        }
+      end
+
+      specify do
+        mock_elasticsearch_response(DummiesIndex, raw_response) do
+          expect(DummiesIndex.query({}).hits).to eq(hits)
+        end
+      end
+    end
+
+    context 'mocks by response sources' do
+      specify do
+        mock_elasticsearch_response_sources(DummiesIndex, sources) do
+          expect(DummiesIndex.query({}).hits).to eq(hits)
+        end
+      end
+    end
+  end
+
+  describe 'assert correct elasticsearch query' do
+    let(:query) do
+      DummiesIndex.filter.should { multi_match foo: 'bar' }.filter { match foo: 'bar' }
+    end
+
+    let(:expected_query) do
+      {
+        index: ['dummies'],
+        body: {
+          query: {
+            bool: {
+              filter: {
+                bool: {
+                  must: {
+                    match: {foo: 'bar'}
+                  },
+                  should: {
+                    multi_match: {foo: 'bar'}
+                  }
+                }
+              }
+            }
+          }
+        }
+      }
+    end
+
+    context 'will be built' do
+      specify do
+        expect { assert_elasticsearch_query(query, expected_query) }.not_to raise_error
+      end
+    end
+
+    context 'will not be built' do
+      let(:unexpected_query) do
+        {
+          index: ['what?'],
+          body: {}
+        }
+      end
+
+      let(:unexpected_query_error_message) do
+        'got {:index=>["dummies"], :body=>{:query=>{:bool=>{:filter=>{:bool=>{:must=>{:match=>{:foo=>"bar"}}, :should=>{:multi_match=>{:foo=>"bar"}}}}}}}} instead of expected query.'
+      end
+
+      specify do
+        expect { assert_elasticsearch_query(query, unexpected_query) }
+          .to raise_error(RuntimeError, unexpected_query_error_message)
+      end
+    end
+  end
+
   context 'assert_indexes' do
     specify 'doesn\'t fail when index updates correctly' do
       expect do

data/spec/chewy/rspec/build_query_spec.rb

@@ -0,0 +1,34 @@
+require 'spec_helper'
+
+describe :build_query do
+  before do
+    stub_model(:city)
+    stub_index(:cities) { index_scope City }
+    CitiesIndex.create
+  end
+
+  let(:expected_query) do
+    {
+      index: ['cities'],
+      body: {
+        query: {
+          match: {name: 'name'}
+        }
+      }
+    }
+  end
+  let(:dummy_query) { {match: {name: 'name'}} }
+  let(:unexpected_query) { {match: {name: 'name'}} }
+
+  context 'build expected query' do
+    specify do
+      expect(CitiesIndex.query(dummy_query)).to build_query(expected_query)
+    end
+  end
+
+  context 'not to build unexpected query' do
+    specify do
+      expect(CitiesIndex.query(dummy_query)).not_to build_query(unexpected_query)
+    end
+  end
+end

data/spec/chewy/rspec/helpers_spec.rb

@@ -0,0 +1,61 @@
+require 'spec_helper'
+
+describe :rspec_helper do
+  include ::Chewy::Rspec::Helpers
+
+  before do
+    stub_model(:city)
+    stub_index(:cities) { index_scope City }
+    CitiesIndex.create
+  end
+
+  let(:hits) do
+    [
+      {
+        '_index' => 'cities',
+        '_type' => '_doc',
+        '_id' => '1',
+        '_score' => 3.14,
+        '_source' => source
+      }
+    ]
+  end
+
+  let(:source) { {'name' => 'some_name'} }
+  let(:sources) { [source] }
+
+  context :mock_elasticsearch_response do
+    let(:raw_response) do
+      {
+        'took' => 4,
+        'timed_out' => false,
+        '_shards' => {
+          'total' => 1,
+          'successful' => 1,
+          'skipped' => 0,
+          'failed' => 0
+        },
+        'hits' => {
+          'total' => {
+            'value' => 1,
+            'relation' => 'eq'
+          },
+          'max_score' => 1.0,
+          'hits' => hits
+        }
+      }
+    end
+
+    specify do
+      mock_elasticsearch_response(CitiesIndex, raw_response)
+      expect(CitiesIndex.query({}).hits).to eq(hits)
+    end
+  end
+
+  context :mock_elasticsearch_response_sources do
+    specify do
+      mock_elasticsearch_response_sources(CitiesIndex, sources)
+      expect(CitiesIndex.query({}).hits).to eq(hits)
+    end
+  end
+end

data/spec/chewy/search/parameters/order_spec.rb

@@ -4,26 +4,29 @@ describe Chewy::Search::Parameters::Order do
   subject { described_class.new(%i[foo bar]) }
 
   describe '#initialize' do
-    specify { expect(described_class.new.value).to eq({}) }
-    specify { expect(described_class.new(nil).value).to eq({}) }
-    specify { expect(described_class.new('').value).to eq({}) }
-    specify { expect(described_class.new(42).value).to eq('42' => nil) }
-    specify { expect(described_class.new([42, 43]).value).to eq('42' => nil, '43' => nil) }
-    specify { expect(described_class.new(a: 1).value).to eq('a' => 1) }
-    specify { expect(described_class.new(['', 43, {a: 1}]).value).to eq('a' => 1, '43' => nil) }
+    specify { expect(described_class.new.value).to eq([]) }
+    specify { expect(described_class.new(nil).value).to eq([]) }
+    specify { expect(described_class.new('').value).to eq([]) }
+    specify { expect(described_class.new(42).value).to eq(['42']) }
+    specify { expect(described_class.new([42, 43]).value).to eq(%w[42 43]) }
+    specify { expect(described_class.new([42, 42]).value).to eq(%w[42 42]) }
+    specify { expect(described_class.new([42, [43, 44]]).value).to eq(%w[42 43 44]) }
+    specify { expect(described_class.new(a: 1).value).to eq([{'a' => 1}]) }
+    specify { expect(described_class.new(['a', {a: 1}, {a: 2}]).value).to eq(['a', {'a' => 1}, {'a' => 2}]) }
+    specify { expect(described_class.new(['', 43, {a: 1}]).value).to eq(['43', {'a' => 1}]) }
   end
 
   describe '#replace!' do
     specify do
       expect { subject.replace!(foo: {}) }
         .to change { subject.value }
-        .from('foo' => nil, 'bar' => nil).to('foo' => {})
+        .from(%w[foo bar]).to([{'foo' => {}}])
     end
 
     specify do
       expect { subject.replace!(nil) }
         .to change { subject.value }
-        .from('foo' => nil, 'bar' => nil).to({})
+        .from(%w[foo bar]).to([])
     end
   end
 
@@ -31,7 +34,7 @@ describe Chewy::Search::Parameters::Order do
     specify do
       expect { subject.update!(foo: {}) }
         .to change { subject.value }
-        .from('foo' => nil, 'bar' => nil).to('foo' => {}, 'bar' => nil)
+        .from(%w[foo bar]).to(['foo', 'bar', {'foo' => {}}])
     end
 
     specify { expect { subject.update!(nil) }.not_to change { subject.value } }
@@ -41,7 +44,7 @@ describe Chewy::Search::Parameters::Order do
     specify do
       expect { subject.merge!(described_class.new(foo: {})) }
         .to change { subject.value }
-        .from('foo' => nil, 'bar' => nil).to('foo' => {}, 'bar' => nil)
+        .from(%w[foo bar]).to(['foo', 'bar', {'foo' => {}}])
     end
 
     specify { expect { subject.merge!(described_class.new) }.not_to change { subject.value } }
@@ -51,6 +54,7 @@ describe Chewy::Search::Parameters::Order do
     specify { expect(described_class.new.render).to be_nil }
     specify { expect(described_class.new(:foo).render).to eq(sort: ['foo']) }
     specify { expect(described_class.new([:foo, {bar: 42}, :baz]).render).to eq(sort: ['foo', {'bar' => 42}, 'baz']) }
+    specify { expect(described_class.new([:foo, {bar: 42}, {bar: 43}, :baz]).render).to eq(sort: ['foo', {'bar' => 42}, {'bar' => 43}, 'baz']) }
   end
 
   describe '#==' do
@@ -59,7 +63,10 @@ describe Chewy::Search::Parameters::Order do
     specify { expect(described_class.new(:foo)).not_to eq(described_class.new(:bar)) }
     specify { expect(described_class.new(%i[foo bar])).to eq(described_class.new(%i[foo bar])) }
     specify { expect(described_class.new(%i[foo bar])).not_to eq(described_class.new(%i[bar foo])) }
+    specify { expect(described_class.new(%i[foo foo])).not_to eq(described_class.new(%i[foo])) }
     specify { expect(described_class.new(foo: {a: 42})).to eq(described_class.new(foo: {a: 42})) }
     specify { expect(described_class.new(foo: {a: 42})).not_to eq(described_class.new(foo: {b: 42})) }
+    specify { expect(described_class.new(['foo', {'foo' => 42}])).not_to eq(described_class.new([{'foo' => 42}, 'foo'])) }
+    specify { expect(described_class.new([{'foo' => 42}, {'foo' => 43}])).not_to eq(described_class.new([{'foo' => 43}, {'foo' => 42}])) }
   end
 end

data/spec/chewy/search/parameters_spec.rb

@@ -13,7 +13,7 @@ describe Chewy::Search::Parameters do
 
     specify { expect(subject.storages[:limit]).to equal(limit) }
     specify { expect(subject.storages[:limit].value).to eq(3) }
-    specify { expect(subject.storages[:order].value).to eq('foo' => nil) }
+    specify { expect(subject.storages[:order].value).to eq(['foo']) }
 
     specify { expect { described_class.new(offset: limit) }.to raise_error(TypeError) }
   end

data/spec/chewy/search/request_spec.rb

@@ -177,7 +177,7 @@ describe Chewy::Search::Request do
   describe '#order' do
     specify { expect(subject.order(:foo).render[:body]).to include(sort: ['foo']) }
     specify { expect(subject.order(foo: 42).order(nil).render[:body]).to include(sort: ['foo' => 42]) }
-    specify { expect(subject.order(foo: 42).order(foo: 43).render[:body]).to include(sort: ['foo' => 43]) }
+    specify { expect(subject.order(foo: 42).order(foo: 43).render[:body]).to include(sort: [{'foo' => 42}, {'foo' => 43}]) }
     specify { expect(subject.order(:foo).order(:bar, :baz).render[:body]).to include(sort: %w[foo bar baz]) }
     specify { expect(subject.order(nil).render[:body]).to be_blank }
     specify { expect { subject.order(:foo) }.not_to change { subject.render } }

data/spec/spec_helper.rb

@@ -8,7 +8,6 @@ require 'rspec/its'
 require 'rspec/collection_matchers'
 
 require 'timecop'
-require 'ruby-progressbar'
 
 Kaminari::Hooks.init if defined?(::Kaminari::Hooks)
 

data/spec/support/active_record.rb

@@ -1,6 +1,6 @@
 require 'database_cleaner'
 
-ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: 'file::memory:?cache=shared', pool: 1)
+ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: 'file::memory:?cache=shared', pool: 10)
 ActiveRecord::Base.logger = Logger.new('/dev/null')
 if ActiveRecord::Base.respond_to?(:raise_in_transactional_callbacks)
   ActiveRecord::Base.raise_in_transactional_callbacks = true

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: chewy
 version: !ruby/object:Gem::Version
-  version: 7.2.1
+  version: 7.2.2
 platform: ruby
 authors:
 - Toptal, LLC
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-05-11 00:00:00.000000000 Z
+date: 2021-05-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: database_cleaner
@@ -185,14 +185,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 6.3.0
+        version: 7.12.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 6.3.0
+        version: 7.12.0
 - !ruby/object:Gem::Dependency
   name: elasticsearch-dsl
   requirement: !ruby/object:Gem::Requirement
@@ -207,20 +207,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: ruby-progressbar
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 description: Chewy provides functionality for Elasticsearch index handling, documents
   import mappings and chainable query DSL
 email:
@@ -272,7 +258,6 @@ files:
 - lib/chewy/index/import/bulk_request.rb
 - lib/chewy/index/import/journal_builder.rb
 - lib/chewy/index/import/routine.rb
-- lib/chewy/index/import/thread_safe_progress_bar.rb
 - lib/chewy/index/mapping.rb
 - lib/chewy/index/observe.rb
 - lib/chewy/index/settings.rb
@@ -290,6 +275,8 @@ files:
 - lib/chewy/rake_helper.rb
 - lib/chewy/repository.rb
 - lib/chewy/rspec.rb
+- lib/chewy/rspec/build_query.rb
+- lib/chewy/rspec/helpers.rb
 - lib/chewy/rspec/update_index.rb
 - lib/chewy/runtime.rb
 - lib/chewy/runtime/version.rb
@@ -379,6 +366,8 @@ files:
 - spec/chewy/multi_search_spec.rb
 - spec/chewy/rake_helper_spec.rb
 - spec/chewy/repository_spec.rb
+- spec/chewy/rspec/build_query_spec.rb
+- spec/chewy/rspec/helpers_spec.rb
 - spec/chewy/rspec/update_index_spec.rb
 - spec/chewy/runtime/version_spec.rb
 - spec/chewy/runtime_spec.rb
@@ -487,6 +476,8 @@ test_files:
 - spec/chewy/multi_search_spec.rb
 - spec/chewy/rake_helper_spec.rb
 - spec/chewy/repository_spec.rb
+- spec/chewy/rspec/build_query_spec.rb
+- spec/chewy/rspec/helpers_spec.rb
 - spec/chewy/rspec/update_index_spec.rb
 - spec/chewy/runtime/version_spec.rb
 - spec/chewy/runtime_spec.rb

data/lib/chewy/index/import/thread_safe_progress_bar.rb

@@ -1,40 +0,0 @@
-module Chewy
-  class Index
-    module Import
-      # This class performs the threading for parallel import to avoid concurrency during progressbar output.
-      #
-      # @see Chewy::Type::Import::ClassMethods#import with `parallel: true` option
-      class ThreadSafeProgressBar
-        def initialize(enabled)
-          @enabled = enabled
-
-          return unless @enabled
-
-          @mutex = Mutex.new
-          @released = false
-          @progressbar = ProgressBar.create total: nil
-          Thread.new do
-            ActiveRecord::Base.connection_pool.with_connection do
-              @mutex.synchronize { @released = true }
-              @progressbar.total = yield
-            end
-          end
-        end
-
-        def increment(value)
-          return unless @enabled
-
-          @mutex.synchronize do
-            @progressbar.progress += value
-          end
-        end
-
-        def wait_until_ready
-          return true unless @enabled
-
-          @mutex.synchronize { @released } until @released
-        end
-      end
-    end
-  end
-end