chewy 7.2.7 → 7.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f720ba38bdda37de9d1a9b54d0b9c28efbe444e3c9cdfb8e3a1fef0f01aeaf85
-  data.tar.gz: 01041cfe59a33c5b9b07bc46f4e5c0d9a68802a7ddf6ad99aebf7a34c85b32b2
+  metadata.gz: 79428eb9af436a7a4c80a2db3c034583768eacc45d00074525f8e6559341ae20
+  data.tar.gz: 23377b8abb6ebfa81f8140990916022f8bbca769b5ad4474a076c8d890ece979
 SHA512:
-  metadata.gz: 01567be06ff1aa7deb0cd9e2459c9a094d80c113e088e1eac7dbdea0cf6b74bbb78579923bf7335bb2114884a3cb3a99ca946009fe3f762566b76a82c17ea7f2
-  data.tar.gz: 0cd235971d30435c1d5ad2b33f70643733fb5af02d7d49891668d216396fdaa0326b7aad4c01bcbf3ed207ef3ee9ac37c02ff269087b9bb6c8e498304dd902d9
+  metadata.gz: 89b6aa83fa54a6b5056623f8255eb34675ad9c3e1494915aa324af752d1a72f9c010fcbcbb3c929a232b32811bbc36956624848efabc09cb382c59fbcee54940
+  data.tar.gz: 90e1da06cf5c38903ca52c6b5379a9271cf277e3c70604b80d77d9ed13496b15ac3874249f2bf67fca7aec3eb9ac9db5e9a20de95eb1da7bb1bfa84307c9c846

data/.github/CODEOWNERS

@@ -1 +1 @@
-.github/workflows @toptal/rogue-one
+.github/workflows @toptal/platform-sre

data/.github/workflows/ruby.yml

@@ -6,7 +6,8 @@ on:
   pull_request:
     types: [
       synchronize, # PR was updated
-      opened # PR was open
+      opened, # PR was open
+      reopened # PR was reopened
     ]
 
 jobs:
@@ -41,7 +42,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        ruby: [ '3.0', 3.1 ]
+        ruby: [ '3.0', '3.1', '3.2' ]
         gemfile: [ rails.6.1.activerecord, rails.7.0.activerecord ]
     name: ${{ matrix.ruby }}-${{ matrix.gemfile }}
 

data/CHANGELOG.md

@@ -8,6 +8,26 @@
 
 ### Bugs Fixed
 
+## 7.3.1 (2023-04-20)
+
+### Bugs Fixed
+
+* [#874](https://github.com/toptal/chewy/pull/874): Fix `chewy:journal:clean` task for ruby 3.x. ([@muk-ai](https://github.com/muk-ai))
+* [#882](https://github.com/toptal/chewy/pull/882): Fix memory leak during `chewy:reset` for ruby 3.2 ([@konalegi](https://github.com/konalegi))
+
+## 7.3.0 (2023-04-03)
+
+### New Features
+
+* [#869](https://github.com/toptal/chewy/pull/869): New strategy - `delayed_sidekiq`. Allow passing `strategy: :delayed_sidekiq` option to `SomeIndex.import([1, ...], strategy: :delayed_sidekiq)`. The strategy is compatible with `update_fields` option as well. ([@skcc321][])
+* [#879](https://github.com/toptal/chewy/pull/879): Configure CI to check for ruby 3.2 compatibility. ([@konalegi][])
+
+### Changes
+
+### Bugs Fixed
+
+* [#856](https://github.com/toptal/chewy/pull/856): Fix return value of subscribed_task_stats used in rake tasks. ([@fabiormoura][])
+
 ## 7.2.7 (2022-11-15)
 
 ### New Features

data/README.md

@@ -5,7 +5,7 @@
 
 # Chewy
 
-Chewy is an ODM (Object Document Mapper), built on top of the [the official Elasticsearch client](https://github.com/elastic/elasticsearch-ruby).
+Chewy is an ODM (Object Document Mapper), built on top of [the official Elasticsearch client](https://github.com/elastic/elasticsearch-ruby).
 
 ## Why Chewy?
 
@@ -458,7 +458,7 @@ field :hierarchy_link, type: :join, relations: {question: %i[answer comment], an
 ```
 assuming you have `comment_type` and `commented_id` fields in your model.
 
-Note that when you reindex a parent, it's children and grandchildren will be reindexed as well.
+Note that when you reindex a parent, its children and grandchildren will be reindexed as well.
 This may require additional queries to the primary database and to elastisearch.
 
 Also note that the join field doesn't support crutches (it should be a field directly defined on the model).
@@ -503,7 +503,7 @@ class ProductsIndex < Chewy::Index
 
   field :name
   # simply use crutch-fetched data as a value:
-  field :category_names, value: ->(product, crutches) { crutches.categories[product.id] }
+  field :category_names, value: ->(product, crutches) { crutches[:categories][product.id] }
 end
 ```
 
@@ -525,7 +525,7 @@ So Chewy Crutches™ technology is able to increase your indexing performance in
 
 ### Witchcraft™ technology
 
-One more experimental technology to increase import performance. As far as you know, chewy defines value proc for every imported field in mapping, so at the import time each of this procs is executed on imported object to extract result document to import. It would be great for performance to use one huge whole-document-returning proc instead. So basically the idea or Witchcraft™ technology is to compile a single document-returning proc from the index definition.
+One more experimental technology to increase import performance. As far as you know, chewy defines value proc for every imported field in mapping, so at the import time each of these procs is executed on imported object to extract result document to import. It would be great for performance to use one huge whole-document-returning proc instead. So basically the idea or Witchcraft™ technology is to compile a single document-returning proc from the index definition.
 
 ```ruby
 index_scope Product
@@ -569,7 +569,7 @@ Obviously not every type of definition might be compiled. There are some restric
   end
   ```
 
-However, it is quite possible that your index definition will be supported by Witchcraft™ technology out of the box in the most of the cases.
+However, it is quite possible that your index definition will be supported by Witchcraft™ technology out of the box in most of the cases.
 
 ### Raw Import
 
@@ -675,7 +675,7 @@ end
 
 You may be wondering why do you need it? The answer is simple: not to lose the data.
 
-Imagine that you reset your index in a zero-downtime manner (to separate index), and at the meantime somebody keeps updating the data frequently (to old index). So all these actions will be written to the journal index and you'll be able to apply them after index reset using the `Chewy::Journal` interface.
+Imagine that you reset your index in a zero-downtime manner (to separate index), and in the meantime somebody keeps updating the data frequently (to old index). So all these actions will be written to the journal index and you'll be able to apply them after index reset using the `Chewy::Journal` interface.
 
 When enabled, journal can grow to enormous size, consider setting up cron job that would clean it occasionally using [`chewy:journal:clean` rake task](#chewyjournal).
 
@@ -774,6 +774,80 @@ The default queue name is `chewy`, you can customize it in settings: `sidekiq.qu
 Chewy.settings[:sidekiq] = {queue: :low}
 ```
 
+#### `:delayed_sidekiq`
+
+It accumulates ids of records to be reindexed during the latency window in redis and then does the reindexing of all accumulated records at once.
+The strategy is very useful in case of frequently mutated records.
+It supports `update_fields` option, so it will try to select just enough data from the DB
+
+There are three options that can be defined in the index:
+```ruby
+class CitiesIndex...
+  strategy_config delayed_sidekiq: {
+    latency: 3,
+    margin: 2,
+    ttl: 60 * 60 * 24,
+    reindex_wrapper: ->(&reindex) {
+      ActiveRecord::Base.connected_to(role: :reading) { reindex.call }
+    }
+    # latency - will prevent scheduling identical jobs
+    # margin - main purpose is to cover db replication lag by the margin
+    # ttl - a chunk expiration time (in seconds)
+    # reindex_wrapper - lambda that accepts block to wrap that reindex process AR connection block.
+  }
+
+  ...
+end
+```
+
+Also you can define defaults in the `initializers/chewy.rb`
+```ruby
+Chewy.settings = {
+  strategy_config: {
+    delayed_sidekiq: {
+      latency: 3,
+      margin: 2,
+      ttl: 60 * 60 * 24,
+      reindex_wrapper: ->(&reindex) {
+        ActiveRecord::Base.connected_to(role: :reading) { reindex.call }
+      }
+    }
+  }
+}
+
+```
+or in `config/chewy.yml`
+```ruby
+  strategy_config:
+    delayed_sidekiq:
+      latency: 3
+      margin: 2
+      ttl: <%= 60 * 60 * 24 %>
+      # reindex_wrapper setting is not possible here!!! use the initializer instead
+```
+
+You can use the strategy identically to other strategies
+```ruby
+Chewy.strategy(:delayed_sidekiq) do
+  City.popular.map(&:do_some_update_action!)
+end
+```
+
+The default queue name is `chewy`, you can customize it in settings: `sidekiq.queue_name`
+```
+Chewy.settings[:sidekiq] = {queue: :low}
+```
+
+Explicit call of the reindex using `:delayed_sidekiq strategy`
+```ruby
+CitiesIndex.import([1, 2, 3], strategy: :delayed_sidekiq)
+```
+
+Explicit call of the reindex using `:delayed_sidekiq` strategy with `:update_fields` support
+```ruby
+CitiesIndex.import([1, 2, 3], update_fields: [:name], strategy: :delayed_sidekiq)
+```
+
 #### `:active_job`
 
 This does the same thing as `:atomic`, but using ActiveJob. This will inherit the ActiveJob configuration settings including the `active_job.queue_adapter` setting for the environment. Patch `Chewy::Strategy::ActiveJob::Worker` for index updates improving.
@@ -810,7 +884,9 @@ It is convenient for use in e.g. the Rails console with non-block notation:
 
 #### `:bypass`
 
-The bypass strategy simply silences index updates.
+When the bypass strategy is active the index will not be automatically updated on object save.
+
+For example, on `City.first.save!` the cities index would not be updated.
 
 #### Nesting
 
@@ -888,7 +964,7 @@ Chewy has notifying the following events:
     {index: 30, delete: 5}
     ```
 
-  * `payload[:errors]`: might not exists. Contains grouped errors with objects ids list:
+  * `payload[:errors]`: might not exist. Contains grouped errors with objects ids list:
 
     ```ruby
     {index: {
@@ -1020,7 +1096,7 @@ Request DSL also provides additional scope actions, like `delete_all`, `exists?`
 
 #### Pagination
 
-The request DSL supports pagination with `Kaminari`. An extension is enabled on initializtion if `Kaminari` is available. See [Chewy::Search](lib/chewy/search.rb) and [Chewy::Search::Pagination::Kaminari](lib/chewy/search/pagination/kaminari.rb) for details.
+The request DSL supports pagination with `Kaminari`. An extension is enabled on initialization if `Kaminari` is available. See [Chewy::Search](lib/chewy/search.rb) and [Chewy::Search::Pagination::Kaminari](lib/chewy/search/pagination/kaminari.rb) for details.
 
 #### Named scopes
 

data/chewy.gemspec

@@ -19,6 +19,7 @@ Gem::Specification.new do |spec| # rubocop:disable Metrics/BlockLength
 
   spec.add_development_dependency 'database_cleaner'
   spec.add_development_dependency 'elasticsearch-extensions'
+  spec.add_development_dependency 'mock_redis'
   spec.add_development_dependency 'rake'
   spec.add_development_dependency 'rspec', '>= 3.7.0'
   spec.add_development_dependency 'rspec-collection_matchers'

data/lib/chewy/index/crutch.rb

@@ -12,13 +12,21 @@ module Chewy
         def initialize(index, collection)
           @index = index
           @collection = collection
-          @index._crutches.each_key do |name|
-            singleton_class.class_eval <<-METHOD, __FILE__, __LINE__ + 1
-              def #{name}
-                @#{name} ||= @index._crutches[:#{name}].call @collection
-              end
-            METHOD
-          end
+          @crutches_instances = {}
+        end
+
+        def method_missing(name, *, **)
+          return self[name] if @index._crutches.key?(name)
+
+          super
+        end
+
+        def respond_to_missing?(name, include_private = false)
+          @index._crutches.key?(name) || super
+        end
+
+        def [](name)
+          @crutches_instances[name] ||= @index._crutches[:"#{name}"].call(@collection)
         end
       end
 

data/lib/chewy/index/import.rb

@@ -73,7 +73,7 @@ module Chewy
         # @option options [true, Integer, Hash] parallel enables parallel import processing with the Parallel gem, accepts the number of workers or any Parallel gem acceptable options
         # @return [true, false] false in case of errors
         ruby2_keywords def import(*args)
-          import_routine(*args).blank?
+          intercept_import_using_strategy(*args).blank?
         end
 
         # @!method import!(*collection, **options)
@@ -84,7 +84,8 @@ module Chewy
         #
         # @raise [Chewy::ImportFailed] in case of errors
         ruby2_keywords def import!(*args)
-          errors = import_routine(*args)
+          errors = intercept_import_using_strategy(*args)
+
           raise Chewy::ImportFailed.new(self, errors) if errors.present?
 
           true
@@ -126,6 +127,32 @@ module Chewy
 
       private
 
+        def intercept_import_using_strategy(*args)
+          args_clone = args.deep_dup
+          options = args_clone.extract_options!
+          strategy = options.delete(:strategy)
+
+          return import_routine(*args) if strategy.blank?
+
+          ids = args_clone.flatten
+          return {} if ids.blank?
+          return {argument: {"#{strategy} supports ids only!" => ids}} unless ids.all? do |id|
+            id.respond_to?(:to_i)
+          end
+
+          case strategy
+          when :delayed_sidekiq
+            begin
+              Chewy::Strategy::DelayedSidekiq::Scheduler.new(self, ids, options).postpone
+              {} # success. errors handling convention
+            rescue StandardError => e
+              {scheduler: {e.message => ids}}
+            end
+          else
+            {argument: {"unsupported strategy: '#{strategy}'" => ids}}
+          end
+        end
+
         def import_routine(*args)
           return if !args.first.nil? && empty_objects_or_scope?(args.first)
 

data/lib/chewy/index.rb

@@ -20,6 +20,10 @@ module Chewy
       pipeline raw_import refresh replication
     ].freeze
 
+    STRATEGY_OPTIONS = {
+      delayed_sidekiq: %i[latency margin ttl reindex_wrapper]
+    }.freeze
+
     include Search
     include Actions
     include Aliases
@@ -221,6 +225,27 @@ module Chewy
         params.assert_valid_keys(IMPORT_OPTIONS_KEYS)
         self._default_import_options = _default_import_options.merge(params)
       end
+
+      def strategy_config(params = {})
+        @strategy_config ||= begin
+          config_struct = Struct.new(*STRATEGY_OPTIONS.keys).new
+
+          STRATEGY_OPTIONS.each_with_object(config_struct) do |(strategy, options), res|
+            res[strategy] = case strategy
+            when :delayed_sidekiq
+              Struct.new(*STRATEGY_OPTIONS[strategy]).new.tap do |config|
+                options.each do |option|
+                  config[option] = params.dig(strategy, option) || Chewy.configuration.dig(:strategy_config, strategy, option)
+                end
+
+                config[:reindex_wrapper] ||= ->(&reindex) { reindex.call } # default wrapper
+              end
+            else
+              raise NotImplementedError, "Unsupported strategy: '#{strategy}'"
+            end
+          end
+        end
+      end
     end
   end
 end

data/lib/chewy/rake_helper.rb

@@ -270,6 +270,7 @@ module Chewy
         ActiveSupport::Notifications.subscribed(JOURNAL_CALLBACK.curry[output], 'apply_journal.chewy') do
           ActiveSupport::Notifications.subscribed(IMPORT_CALLBACK.curry[output], 'import_objects.chewy', &block)
         end
+      ensure
         output.puts "Total: #{human_duration(Time.now - start)}"
       end
 

data/lib/chewy/strategy/delayed_sidekiq/scheduler.rb

@@ -0,0 +1,148 @@
+# frozen_string_literal: true
+
+require_relative '../../index'
+
+# The class is responsible for accumulating in redis [type, ids]
+# that were requested to be reindexed during `latency` seconds.
+# The reindex job is going to be scheduled after a `latency` seconds.
+# that job is going to read accumulated [type, ids] from the redis
+# and reindex all them at once.
+module Chewy
+  class Strategy
+    class DelayedSidekiq
+      require_relative 'worker'
+
+      class Scheduler
+        DEFAULT_TTL = 60 * 60 * 24 # in seconds
+        DEFAULT_LATENCY = 10
+        DEFAULT_MARGIN = 2
+        DEFAULT_QUEUE = 'chewy'
+        KEY_PREFIX = 'chewy:delayed_sidekiq'
+        FALLBACK_FIELDS = 'all'
+        FIELDS_IDS_SEPARATOR = ';'
+        IDS_SEPARATOR = ','
+
+        def initialize(type, ids, options = {})
+          @type = type
+          @ids = ids
+          @options = options
+        end
+
+        # the diagram:
+        #
+        #  inputs:
+        #  latency == 2
+        #  reindex_time = Time.current
+        #
+        #  Parallel OR Sequential triggers of reindex:          |  What is going on in reindex store (Redis):
+        #  --------------------------------------------------------------------------------------------------
+        #                                                       |
+        #  process 1 (reindex_time):                            |  chewy:delayed_sidekiq:CitiesIndex:1679347866 = [1]
+        #    Schedule.new(CitiesIndex, [1]).postpone            |  chewy:delayed_sidekiq:timechunks = [{ score: 1679347866, "chewy:delayed_sidekiq:CitiesIndex:1679347866"}]
+        #                                                       |  & schedule a DelayedSidekiq::Worker at 1679347869 (at + 3)
+        #                                                       |    it will zpop chewy:delayed_sidekiq:timechunks up to 1679347866 score and reindex all ids with zpoped keys
+        #                                                       |      chewy:delayed_sidekiq:CitiesIndex:1679347866
+        #                                                       |
+        #                                                       |
+        #  process 2 (reindex_time):                            |  chewy:delayed_sidekiq:CitiesIndex:1679347866 = [1, 2]
+        #    Schedule.new(CitiesIndex, [2]).postpone            |  chewy:delayed_sidekiq:timechunks = [{ score: 1679347866, "chewy:delayed_sidekiq:CitiesIndex:1679347866"}]
+        #                                                       |  & do not schedule a new worker
+        #                                                       |
+        #                                                       |
+        #  process 1 (reindex_time + (latency - 1).seconds):    |  chewy:delayed_sidekiq:CitiesIndex:1679347866 = [1, 2, 3]
+        #    Schedule.new(CitiesIndex, [3]).postpone            |  chewy:delayed_sidekiq:timechunks = [{ score: 1679347866, "chewy:delayed_sidekiq:CitiesIndex:1679347866"}]
+        #                                                       |  & do not schedule a new worker
+        #                                                       |
+        #                                                       |
+        #  process 2 (reindex_time + (latency + 1).seconds):    |  chewy:delayed_sidekiq:CitiesIndex:1679347866 = [1, 2, 3]
+        #    Schedule.new(CitiesIndex, [4]).postpone            |  chewy:delayed_sidekiq:CitiesIndex:1679347868 = [4]
+        #                                                       |  chewy:delayed_sidekiq:timechunks = [
+        #                                                       |    { score: 1679347866, "chewy:delayed_sidekiq:CitiesIndex:1679347866"}
+        #                                                       |    { score: 1679347868, "chewy:delayed_sidekiq:CitiesIndex:1679347868"}
+        #                                                       |  ]
+        #                                                       |  & schedule a DelayedSidekiq::Worker at 1679347871 (at + 3)
+        #                                                       |    it will zpop chewy:delayed_sidekiq:timechunks up to 1679347868 score and reindex all ids with zpoped keys
+        #                                                       |      chewy:delayed_sidekiq:CitiesIndex:1679347866 (in case of failed previous reindex),
+        #                                                       |      chewy:delayed_sidekiq:CitiesIndex:1679347868
+        def postpone
+          ::Sidekiq.redis do |redis|
+            # warning: Redis#sadd will always return an Integer in Redis 5.0.0. Use Redis#sadd? instead
+            if redis.respond_to?(:sadd?)
+              redis.sadd?(timechunk_key, serialize_data)
+            else
+              redis.sadd(timechunk_key, serialize_data)
+            end
+
+            redis.expire(timechunk_key, ttl)
+
+            unless redis.zrank(timechunks_key, timechunk_key)
+              redis.zadd(timechunks_key, at, timechunk_key)
+              redis.expire(timechunks_key, ttl)
+
+              ::Sidekiq::Client.push(
+                'queue' => sidekiq_queue,
+                'at' => at + margin,
+                'class' => Chewy::Strategy::DelayedSidekiq::Worker,
+                'args' => [type_name, at]
+              )
+            end
+          end
+        end
+
+      private
+
+        attr_reader :type, :ids, :options
+
+        # this method returns predictable value that jumps by latency value
+        # another words each latency seconds it return the same value
+        def at
+          @at ||= begin
+            schedule_at = latency.seconds.from_now.to_f
+
+            (schedule_at - (schedule_at % latency)).to_i
+          end
+        end
+
+        def fields
+          options[:update_fields].presence || [FALLBACK_FIELDS]
+        end
+
+        def timechunks_key
+          "#{KEY_PREFIX}:#{type_name}:timechunks"
+        end
+
+        def timechunk_key
+          "#{KEY_PREFIX}:#{type_name}:#{at}"
+        end
+
+        def serialize_data
+          [ids.join(IDS_SEPARATOR), fields.join(IDS_SEPARATOR)].join(FIELDS_IDS_SEPARATOR)
+        end
+
+        def type_name
+          type.name
+        end
+
+        def latency
+          strategy_config.latency || DEFAULT_LATENCY
+        end
+
+        def margin
+          strategy_config.margin || DEFAULT_MARGIN
+        end
+
+        def ttl
+          strategy_config.ttl || DEFAULT_TTL
+        end
+
+        def sidekiq_queue
+          Chewy.settings.dig(:sidekiq, :queue) || DEFAULT_QUEUE
+        end
+
+        def strategy_config
+          type.strategy_config.delayed_sidekiq
+        end
+      end
+    end
+  end
+end

data/lib/chewy/strategy/delayed_sidekiq/worker.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+module Chewy
+  class Strategy
+    class DelayedSidekiq
+      class Worker
+        include ::Sidekiq::Worker
+
+        def perform(type, score, options = {})
+          options[:refresh] = !Chewy.disable_refresh_async if Chewy.disable_refresh_async
+
+          ::Sidekiq.redis do |redis|
+            timechunks_key = "#{Scheduler::KEY_PREFIX}:#{type}:timechunks"
+            timechunk_keys = redis.zrangebyscore(timechunks_key, -1, score)
+            members = timechunk_keys.flat_map { |timechunk_key| redis.smembers(timechunk_key) }.compact
+
+            # extract ids and fields & do the reset of records
+            ids, fields = extract_ids_and_fields(members)
+            options[:update_fields] = fields if fields
+
+            index = type.constantize
+            index.strategy_config.delayed_sidekiq.reindex_wrapper.call do
+              options.any? ? index.import!(ids, **options) : index.import!(ids)
+            end
+
+            redis.del(timechunk_keys)
+            redis.zremrangebyscore(timechunks_key, -1, score)
+          end
+        end
+
+      private
+
+        def extract_ids_and_fields(members)
+          ids = []
+          fields = []
+
+          members.each do |member|
+            member_ids, member_fields = member.split(Scheduler::FIELDS_IDS_SEPARATOR).map do |v|
+              v.split(Scheduler::IDS_SEPARATOR)
+            end
+            ids |= member_ids
+            fields |= member_fields
+          end
+
+          fields = nil if fields.include?(Scheduler::FALLBACK_FIELDS)
+
+          [ids.map(&:to_i), fields]
+        end
+      end
+    end
+  end
+end

data/lib/chewy/strategy/delayed_sidekiq.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module Chewy
+  class Strategy
+    class DelayedSidekiq < Sidekiq
+      require_relative 'delayed_sidekiq/scheduler'
+
+      def leave
+        @stash.each do |type, ids|
+          next if ids.empty?
+
+          DelayedSidekiq::Scheduler.new(type, ids).postpone
+        end
+      end
+    end
+  end
+end

data/lib/chewy/strategy.rb

@@ -8,6 +8,7 @@ begin
   require 'sidekiq'
   require 'chewy/strategy/sidekiq'
   require 'chewy/strategy/lazy_sidekiq'
+  require 'chewy/strategy/delayed_sidekiq'
 rescue LoadError
   nil
 end

data/lib/chewy/version.rb

@@ -1,3 +1,3 @@
 module Chewy
-  VERSION = '7.2.7'.freeze
+  VERSION = '7.3.1'.freeze
 end

data/lib/tasks/chewy.rake

@@ -96,7 +96,7 @@ namespace :chewy do
     task clean: :environment do |_task, args|
       delete_options = Chewy::RakeHelper.delete_by_query_options_from_env(ENV)
       Chewy::RakeHelper.journal_clean(
-        [
+        **[
           parse_journal_args(args.extras),
           {delete_by_query_options: delete_options}
         ].reduce({}, :merge)

data/spec/chewy/rake_helper_spec.rb

@@ -1,4 +1,5 @@
 require 'spec_helper'
+require 'rake'
 
 describe Chewy::RakeHelper, :orm do
   before { Chewy.massacre }
@@ -456,6 +457,17 @@ Total: \\d+s\\Z
 Total: \\d+s\\Z
       OUTPUT
     end
+
+    context 'execute "chewy:journal:clean" rake task' do
+      subject(:task) { Rake.application['chewy:journal:clean'] }
+      before do
+        Rake::DefaultLoader.new.load('lib/tasks/chewy.rake')
+        Rake::Task.define_task(:environment)
+      end
+      it 'does not raise error' do
+        expect { task.invoke }.to_not raise_error
+      end
+    end
   end
 
   describe '.reindex' do
@@ -570,4 +582,11 @@ Total: \\d+s\\Z
       end
     end
   end
+
+  describe '.subscribed_task_stats' do
+    specify do
+      block_output = described_class.subscribed_task_stats(StringIO.new) { 'expected output' }
+      expect(block_output).to eq('expected output')
+    end
+  end
 end

data/spec/chewy/strategy/delayed_sidekiq_spec.rb

@@ -0,0 +1,190 @@
+require 'spec_helper'
+
+if defined?(Sidekiq)
+  require 'sidekiq/testing'
+  require 'mock_redis'
+
+  describe Chewy::Strategy::DelayedSidekiq do
+    around do |example|
+      Chewy.strategy(:bypass) { example.run }
+    end
+
+    before do
+      redis = MockRedis.new
+      allow(Sidekiq).to receive(:redis).and_yield(redis)
+      Sidekiq::Worker.clear_all
+    end
+
+    before do
+      stub_model(:city) do
+        update_index('cities') { self }
+      end
+
+      stub_index(:cities) do
+        index_scope City
+      end
+    end
+
+    let(:city) { City.create!(name: 'hello') }
+    let(:other_city) { City.create!(name: 'world') }
+
+    it 'does not trigger immediate reindex due to it`s async nature' do
+      expect { [city, other_city].map(&:save!) }
+        .not_to update_index(CitiesIndex, strategy: :delayed_sidekiq)
+    end
+
+    it "respects 'refresh: false' options" do
+      allow(Chewy).to receive(:disable_refresh_async).and_return(true)
+      expect(CitiesIndex).to receive(:import!).with([city.id, other_city.id], refresh: false)
+      scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [city.id, other_city.id])
+      scheduler.postpone
+      Chewy::Strategy::DelayedSidekiq::Worker.drain
+    end
+
+    context 'with default config' do
+      it 'does schedule a job that triggers reindex with default options' do
+        Timecop.freeze do
+          expect(Sidekiq::Client).to receive(:push).with(
+            hash_including(
+              'queue' => 'chewy',
+              'at' => (Time.current.to_i.ceil(-1) + 2.seconds).to_i,
+              'class' => Chewy::Strategy::DelayedSidekiq::Worker,
+              'args' => ['CitiesIndex', an_instance_of(Integer)]
+            )
+          ).and_call_original
+
+          expect($stdout).not_to receive(:puts)
+
+          Sidekiq::Testing.inline! do
+            expect { [city, other_city].map(&:save!) }
+              .to update_index(CitiesIndex, strategy: :delayed_sidekiq)
+              .and_reindex(city, other_city).only
+          end
+        end
+      end
+    end
+
+    context 'with custom config' do
+      before do
+        CitiesIndex.strategy_config(
+          delayed_sidekiq: {
+            reindex_wrapper: lambda { |&reindex|
+              puts 'hello'
+              reindex.call
+            },
+            margin: 5,
+            latency: 60
+          }
+        )
+      end
+
+      it 'respects :strategy_config options' do
+        Timecop.freeze do
+          expect(Sidekiq::Client).to receive(:push).with(
+            hash_including(
+              'queue' => 'chewy',
+              'at' => (60.seconds.from_now.change(sec: 0) + 5.seconds).to_i,
+              'class' => Chewy::Strategy::DelayedSidekiq::Worker,
+              'args' => ['CitiesIndex', an_instance_of(Integer)]
+            )
+          ).and_call_original
+
+          expect($stdout).to receive(:puts).with('hello') # check that reindex_wrapper works
+
+          Sidekiq::Testing.inline! do
+            expect { [city, other_city].map(&:save!) }
+              .to update_index(CitiesIndex, strategy: :delayed_sidekiq)
+              .and_reindex(city, other_city).only
+          end
+        end
+      end
+    end
+
+    context 'two reindex call within the timewindow' do
+      it 'accumulates all ids does the reindex one time' do
+        Timecop.freeze do
+          expect(CitiesIndex).to receive(:import!).with([other_city.id, city.id]).once
+          scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [city.id])
+          scheduler.postpone
+          scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [other_city.id])
+          scheduler.postpone
+          Chewy::Strategy::DelayedSidekiq::Worker.drain
+        end
+      end
+
+      context 'one call with update_fields another one without update_fields' do
+        it 'does reindex of all fields' do
+          Timecop.freeze do
+            expect(CitiesIndex).to receive(:import!).with([other_city.id, city.id]).once
+            scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [city.id], update_fields: ['name'])
+            scheduler.postpone
+            scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [other_city.id])
+            scheduler.postpone
+            Chewy::Strategy::DelayedSidekiq::Worker.drain
+          end
+        end
+      end
+
+      context 'both calls with different update fields' do
+        it 'deos reindex with union of fields' do
+          Timecop.freeze do
+            expect(CitiesIndex).to receive(:import!).with([other_city.id, city.id], update_fields: %w[description name]).once
+            scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [city.id], update_fields: ['name'])
+            scheduler.postpone
+            scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [other_city.id], update_fields: ['description'])
+            scheduler.postpone
+            Chewy::Strategy::DelayedSidekiq::Worker.drain
+          end
+        end
+      end
+    end
+
+    context 'two calls within different timewindows' do
+      it 'does two separate reindexes' do
+        Timecop.freeze do
+          expect(CitiesIndex).to receive(:import!).with([city.id]).once
+          expect(CitiesIndex).to receive(:import!).with([other_city.id]).once
+          Timecop.travel(20.seconds.ago) do
+            scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [city.id])
+            scheduler.postpone
+          end
+          scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [other_city.id])
+          scheduler.postpone
+          Chewy::Strategy::DelayedSidekiq::Worker.drain
+        end
+      end
+    end
+
+    context 'first call has update_fields' do
+      it 'does first reindex with the expected update_fields and second without update_fields' do
+        Timecop.freeze do
+          expect(CitiesIndex).to receive(:import!).with([city.id], update_fields: ['name']).once
+          expect(CitiesIndex).to receive(:import!).with([other_city.id]).once
+          Timecop.travel(20.seconds.ago) do
+            scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [city.id], update_fields: ['name'])
+            scheduler.postpone
+          end
+          scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [other_city.id])
+          scheduler.postpone
+          Chewy::Strategy::DelayedSidekiq::Worker.drain
+        end
+      end
+    end
+
+    context 'both calls have update_fields option' do
+      it 'does both reindexes with their expected update_fields option' do
+        Timecop.freeze do
+          expect(CitiesIndex).to receive(:import!).with([city.id], update_fields: ['name']).once
+          expect(CitiesIndex).to receive(:import!).with([other_city.id], update_fields: ['description']).once
+          Timecop.travel(20.seconds.ago) do
+            scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [city.id], update_fields: ['name'])
+            scheduler.postpone
+          end
+          scheduler = Chewy::Strategy::DelayedSidekiq::Scheduler.new(CitiesIndex, [other_city.id], update_fields: ['description'])
+          scheduler.postpone
+          Chewy::Strategy::DelayedSidekiq::Worker.drain
+        end
+      end
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: chewy
 version: !ruby/object:Gem::Version
-  version: 7.2.7
+  version: 7.3.1
 platform: ruby
 authors:
 - Toptal, LLC
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-11-15 00:00:00.000000000 Z
+date: 2023-04-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: database_cleaner
@@ -39,6 +39,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: mock_redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -344,6 +358,9 @@ files:
 - lib/chewy/strategy/atomic_no_refresh.rb
 - lib/chewy/strategy/base.rb
 - lib/chewy/strategy/bypass.rb
+- lib/chewy/strategy/delayed_sidekiq.rb
+- lib/chewy/strategy/delayed_sidekiq/scheduler.rb
+- lib/chewy/strategy/delayed_sidekiq/worker.rb
 - lib/chewy/strategy/lazy_sidekiq.rb
 - lib/chewy/strategy/sidekiq.rb
 - lib/chewy/strategy/urgent.rb
@@ -437,6 +454,7 @@ files:
 - spec/chewy/strategy/active_job_spec.rb
 - spec/chewy/strategy/atomic_no_refresh_spec.rb
 - spec/chewy/strategy/atomic_spec.rb
+- spec/chewy/strategy/delayed_sidekiq_spec.rb
 - spec/chewy/strategy/lazy_sidekiq_spec.rb
 - spec/chewy/strategy/sidekiq_spec.rb
 - spec/chewy/strategy_spec.rb
@@ -464,7 +482,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.33
+rubygems_version: 3.3.26
 signing_key:
 specification_version: 4
 summary: Elasticsearch ODM client wrapper
@@ -554,6 +572,7 @@ test_files:
 - spec/chewy/strategy/active_job_spec.rb
 - spec/chewy/strategy/atomic_no_refresh_spec.rb
 - spec/chewy/strategy/atomic_spec.rb
+- spec/chewy/strategy/delayed_sidekiq_spec.rb
 - spec/chewy/strategy/lazy_sidekiq_spec.rb
 - spec/chewy/strategy/sidekiq_spec.rb
 - spec/chewy/strategy_spec.rb