sidekiq_bulk_job 0.1.1 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: 1acd50d98090657e0c060fd63d2d1733214b964447a233b7a2a4a83b65e54db2
-  data.tar.gz: be8c7002b9200fcf374067ff49aa0208866ba50f218beedc91b021c42ecfa2b4
+SHA1:
+  metadata.gz: ca8157b27f077c0db69ac593ad1fcbd8c14e0eab
+  data.tar.gz: 6c87da78b2105230eb522ff3dc6c09f4c73c3540
 SHA512:
-  metadata.gz: e9b0fee851c8445ef0bc811a16f95716b5c772d3542b83be95c24c60246c0ea91cae491f1b428f8681cfba9bf3464c032779af4cc14664a550ace37fec1d3bff
-  data.tar.gz: bcf7ceea6a57f54c414c4a1194b5feb5880aaa3fabbe48f72d95016b3bdce059652cb21ed1e511363d56d17519e78bc0075be9d06b0e143d4709fab4f050f811
+  metadata.gz: 5ba3674183b7f3fa818d4c2bec3f8b6d3f8a632afcd98d38a3beff78e165ad8fe2078d0149dc7601e6e78ec80914f10cba35f819707f42f45cef756fe39cf546
+  data.tar.gz: afe1b3f3787cfe828da123984323cb0b45d4877388238b7b080e8b5ea33d1e718e5225ac6f32e8467a2f77dc7340fd791ad17a3e22e8166d3c9a0c03bf8ca3b7

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    sidekiq_bulk_job (0.1.0)
+    sidekiq_bulk_job (0.1.4)
       sidekiq (~> 5.2.7)
 
 GEM

data/README.md

@@ -1,8 +1,6 @@
 # SidekiqBulkJob
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/sidekiq_bulk_job`. To experiment with that code, run `bin/console` for an interactive prompt.
-
-TODO: Delete this and the text above, and describe your gem
+A tool to collect the same class jobs together and running in batch.
 
 ## Installation
 
@@ -22,26 +20,226 @@ Or install it yourself as:
 
 ## Usage
 
-###
+### Initialization：
+
+##### Parameters：
+
+* redis: redis client.
+* logger: log object，default Logger.new(STDOUT).
+* process_fail: a callback when the job fail.
+* async_delay: await delay time，default 60 seconds.
+* scheduled_delay: scheduled job delay time，default 10 seconds.
+* queue: default sidekiq running queue. By default the batch job will run at queue as same as sidekiq worker defined.
+* batch_size: batch size in same job，default 3000.
+* prefix: redis key prefix, default SidekiqBulkJob.
+
+```ruby
+	process_fail = lambda do |job_class_name, args, exception|
+	# do something
+	# send email
+	end
+	SidekiqBulkJob.config({
+		redis: Redis.new,
+		logger: Logger.new(STDOUT),
+		process_fail: process_fail,
+		async_delay: ASYNC_DELAY,
+		scheduled_delay: SCHEDULED_DELAY,
+		queue: :test,
+		batch_size: BATCH_SIZE,
+		prefix: "SidekiqBulkJob"
+	})
+	  // push a job
+	SidekiqBulkJob.perform_async(TestJob, 10)
+```
+
+### Usage
+
+At first define a TestJob as example
 ```ruby
-  process_fail = lambda do |job_class_name, args, exception|
-    # do somethine
-    # send email
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
   end
-  client = Redis.new
-  logger = Logger.new(STDOUT)
-  logger.level = Logger::WARN
-  SidekiqBulkJob.config redis: client, logger: logger, process_fail: process_fail, queue: :default, batch_size: 3000, prefix: "SidekiqBulkJob"
+  ```
+  
+##### Use SidekiqBulkJob async
 
-  // push a job
+  SidekiqBulkJob will collect the same job in to a list, a batch job will create when beyond the ```batch_size``` in  ```async_delay``` amount, and clear the list. The list will continue to collect the job which pushing inside. If reach the```async_delay``` time, the SidekiqBulkJob will also created to finish all job collected.
+  
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+
+  # simple use
   SidekiqBulkJob.perform_async(TestJob, 10)
+
+  # here will not create 1001 job in sidekiq
+  # now there are tow jobs created, one is collected 1000 TestJob in batch, another has 1 job inside.
+  (BATCH_SIZE + 1).times do |i|
+      SidekiqBulkJob.perform_async(TestJob, i)
+  end
 ```
 
-## Development
+##### Use SidekiqWork batch_perform_async to run async task
+
+```ruby
+# same as SidekiqBulkJob.perform_async(TestJob, 10)
+TestJob.batch_perform_async(10)
+```
+
+##### Use SidekiqBulkJob perform_at/perform_in to set scheduled task
+
+```ruby
+# run at 1 minute after with single job
+SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+# same as below
+SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+```
+
+##### Use SidekiqWork batch_perform_at/batch_perform_in to set scheduled task
+
+```ruby
+# same as SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+TestJob.batch_perform_at(1.minutes.after, 10)
+# same as SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+TestJob.batch_perform_in(1.minute, 10)
+```
+
+##### Use setter to set task
+
+```ruby
+# set queue to test and run async
+TestJob.set(queue: :test).batch_perform_async(10)
+# set queue to test and run after 90 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_async(10)
+
+#batch_perform_in first params interval will be overrided 'in'/'at' option at setter
+# run after 90 seconds instead of 10 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_in(10, 10)
+```
+
+## 中文
+
+### 初始化：
+
+##### 参数：
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+* redis: redis client
+* logger: 日志对象，默认Logger.new(STDOUT)
+* process_fail: 当job处理失败的通用回调
+* async_delay: 延迟等待时间，默认60秒
+* scheduled_delay: 定时任务延迟时间，默认10秒
+* queue: 默认运行队列。根据job本身设置的队列运行，当没有设置时候就使用这里设置的队列运行
+* batch_size: 同种类型job批量运行数量，默认3000
+* prefix: 存储到redis的前缀，默认SidekiqBulkJob
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+```ruby
+	process_fail = lambda do |job_class_name, args, exception|
+	# do something
+	# send email
+	end
+	SidekiqBulkJob.config({
+		redis: Redis.new,
+		logger: Logger.new(STDOUT),
+		process_fail: process_fail,
+		async_delay: ASYNC_DELAY,
+		scheduled_delay: SCHEDULED_DELAY,
+		queue: :test,
+		batch_size: BATCH_SIZE,
+		prefix: "SidekiqBulkJob"
+	})
+	  // push a job
+	SidekiqBulkJob.perform_async(TestJob, 10)
+```
+
+### 用法
+
+设置一个TestJob举例子
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+  ```
+  
+##### 使用SidekiqBulkJob的async接口
+
+SidekiqBulkJob会把同类型的job汇总到一个list中，当```async_delay```时间内超过```batch_size```，会新建一个batch job立刻执行汇总的全部jobs，清空list，清空的list会继续收集后续推入的job；如果在```async_delay```时间内未到达```batch_size```则会在最后一个job推入后等待```async_delay```时间创建一个batch job执行汇总的全部jobs
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+
+  # simple use
+  SidekiqBulkJob.perform_async(TestJob, 10)
+
+  # here will not create 1001 job in sidekiq
+  # now there are tow jobs created, one is collected 1000 TestJob in batch, another has 1 job inside.
+  (BATCH_SIZE + 1).times do |i|
+      SidekiqBulkJob.perform_async(TestJob, i)
+  end
+```
+
+##### 使用SidekiqWork的batch_perform_async接口异步执行任务
+
+```ruby
+# same as SidekiqBulkJob.perform_async(TestJob, 10)
+TestJob.batch_perform_async(10)
+```
+
+##### 使用SidekiqBulkJob的perform_at/perform_in接口设置定时任务
+
+```ruby
+# run at 1 minute after with single job
+SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+# same as below
+SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+```
+
+##### 使用SidekiqWork的batch_perform_at/batch_perform_in接口设置定时任务
+
+```ruby
+# same as SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+TestJob.batch_perform_at(1.minutes.after, 10)
+# same as SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+TestJob.batch_perform_in(1.minute, 10)
+```
+
+##### 使用setter设置
+
+```ruby
+# set queue to test and run async
+TestJob.set(queue: :test).batch_perform_async(10)
+# set queue to test and run after 90 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_async(10)
+
+#batch_perform_in first params interval will be overrided 'in'/'at' option at setter
+# run after 90 seconds instead of 10 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_in(10, 10)
+```
 
 ## Contributing
 

data/lib/sidekiq_bulk_job.rb

@@ -23,13 +23,13 @@ module SidekiqBulkJob
     end
 
     def perform_async(job_class, *args)
-      options = Utils.symbolize_keys(@opts)
+      options = SidekiqBulkJob::Utils.symbolize_keys(@opts)
       if options[:at].nil? && options[:in].nil?
         payload = {
             job_class_name: job_class.to_s,
             perfrom_args: args,
             queue: options[:queue] || SidekiqBulkJob.queue
-          }.compact
+          }.select { |_, value| !value.nil? }
         SidekiqBulkJob.process payload
       else
         perform_in(options[:at] || options[:in], job_class, *args)
@@ -45,13 +45,13 @@ module SidekiqBulkJob
 
       # Optimization to enqueue something now that is scheduled to go out now or in the past
       if ts > now.to_f
-        options = Utils.symbolize_keys(@opts)
+        options = SidekiqBulkJob::Utils.symbolize_keys(@opts)
         payload = {
           job_class_name: job_class.to_s,
           at: ts,
           perfrom_args: args,
           queue: options[:queue] || SidekiqBulkJob.queue
-        }.compact
+        }.select { |_, value| !value.nil? }
         SidekiqBulkJob.process payload
       else
         perform_async(job_class, *args)
@@ -64,18 +64,37 @@ module SidekiqBulkJob
 
   class << self
 
-    attr_accessor :prefix, :redis, :queue, :batch_size, :logger, :process_fail
+    attr_accessor :prefix, :redis, :queue, :scheduled_delay, :async_delay, :batch_size, :logger, :process_fail
 
-    def config(redis: , logger: , process_fail: , queue: :default, batch_size: 3000, prefix: "SidekiqBulkJob")
+    def config(redis: , logger: , process_fail: , async_delay: 60, scheduled_delay: 10, queue: :default, batch_size: 3000, prefix: "SidekiqBulkJob")
       if redis.nil?
         raise ArgumentError.new("redis not allow nil")
       end
+      if logger.nil?
+        raise ArgumentError.new("logger not allow nil")
+      end
+      if process_fail.nil?
+        raise ArgumentError.new("process_fail not allow nil")
+      end
+      if async_delay.to_f < 2
+        raise ArgumentError.new("async_delay not allow less than 2 seconds.")
+      elsif async_delay.to_f > 5 * 60
+        raise ArgumentError.new("async_delay not allow greater than 5 minutes.")
+      end
+      if scheduled_delay.to_f < 2
+        raise ArgumentError.new("scheduled_delay not allow less than 2 seconds.")
+      elsif scheduled_delay.to_f > 30
+        raise ArgumentError.new("scheduled_delay not allow greater than 2 seconds.")
+      end
+
       self.redis = redis
       self.queue = queue
       self.batch_size = batch_size
       self.prefix = prefix
       self.logger = logger
       self.process_fail = process_fail
+      self.async_delay = async_delay.to_f
+      self.scheduled_delay = scheduled_delay.to_f
     end
 
     def set(options)
@@ -113,7 +132,7 @@ module SidekiqBulkJob
     def process(job_class_name: , at: nil, perfrom_args: [], queue: self.queue)
       if at.nil?
         key = generate_key(job_class_name)
-        client.lpush key, perfrom_args.to_json
+        client.lpush key, SidekiqBulkJob::Utils.dump(perfrom_args)
         bulk_run(job_class_name, key, queue: queue) if need_flush?(key)
         monitor(job_class_name, queue: queue)
       else
@@ -121,18 +140,18 @@ module SidekiqBulkJob
         args_redis_key = nil
         target = scheduled_set.find do |job|
           if job.klass == SidekiqBulkJob::ScheduledJob.to_s &&
-                job.at.to_i.between?((at - 5).to_i, (at + 30).to_i) # 允许30秒延迟
+                job.at.to_i.between?((at - self.scheduled_delay).to_i, (at + self.scheduled_delay).to_i) # 允许30秒延迟
               _job_class_name, args_redis_key = job.args
               _job_class_name == job_class_name
            end
         end
         if !target.nil? && !args_redis_key.nil? && !args_redis_key.empty?
           # 往现有的job参数set里增加参数
-          client.lpush args_redis_key, perfrom_args.to_json
+          client.lpush args_redis_key, SidekiqBulkJob::Utils.dump(perfrom_args)
         else
           # 新增加一个
           args_redis_key = SecureRandom.hex
-          client.lpush args_redis_key, perfrom_args.to_json
+          client.lpush args_redis_key, SidekiqBulkJob::Utils.dump(perfrom_args)
           SidekiqBulkJob::ScheduledJob.client_push("queue" => queue, "class" => SidekiqBulkJob::ScheduledJob, "at" => at, "args" => [job_class_name, args_redis_key])
         end
       end
@@ -197,7 +216,7 @@ module SidekiqBulkJob
       if !_monitor.nil?
         # TODO debug log
       else
-        SidekiqBulkJob::Monitor.client_push("queue" => queue, "at" => (time_now + 60).to_f, "class" => SidekiqBulkJob::Monitor, "args" => [time_now.to_f, job_class_name])
+        SidekiqBulkJob::Monitor.client_push("queue" => queue, "at" => (time_now + self.async_delay).to_f, "class" => SidekiqBulkJob::Monitor, "args" => [time_now.to_f, job_class_name])
       end
     end
 

data/lib/sidekiq_bulk_job/bulk_error_handler.rb

@@ -0,0 +1,59 @@
+module SidekiqBulkJob
+  class BulkErrorHandler
+
+    ErrorCollection = Struct.new(:args, :exception) do
+      def message
+        exception.message
+      end
+
+      def backtrace
+        exception.backtrace
+      end
+    end
+
+    attr_accessor :job_class_name, :errors, :jid
+
+    def initialize(job_class_name, jid)
+      @jid = jid
+      @job_class_name = job_class_name
+      @errors = []
+    end
+
+    def add(job_args, exception)
+      errors << ErrorCollection.new(job_args, exception)
+    end
+
+    def backtrace
+      errors.map(&:backtrace).flatten
+    end
+
+    def args
+      errors.map(&:args)
+    end
+
+    def failed?
+      !errors.empty?
+    end
+
+    def raise_error
+      error = BulkError.new(errors.map(&:message).join('; '))
+      error.set_backtrace self.backtrace
+      error
+    end
+
+    def retry_count
+      SidekiqBulkJob.redis.incr jid
+    end
+
+    def clear
+      SidekiqBulkJob.redis.del jid
+    end
+
+    class BulkError < StandardError
+      def initialize(message)
+        super(message)
+      end
+    end
+
+  end
+end

data/lib/sidekiq_bulk_job/bulk_job.rb

@@ -1,25 +1,37 @@
 require "sidekiq"
 
 require "sidekiq_bulk_job/job_retry"
+require "sidekiq_bulk_job/bulk_error_handler"
 require "sidekiq_bulk_job/utils"
 
 module SidekiqBulkJob
   class BulkJob
     include Sidekiq::Worker
-    sidekiq_options queue: :default, retry: false
+    sidekiq_options queue: :default, retry: true
 
     def perform(job_class_name, args_array)
-      job = Utils.constantize(job_class_name)
+      target_name, method_name = SidekiqBulkJob::Utils.split_class_name_with_method job_class_name
+      job = SidekiqBulkJob::Utils.constantize(target_name)
+      error_handle = BulkErrorHandler.new(job_class_name, self.jid)
       args_array.each do |_args|
         begin
-          args = JSON.parse _args
-          job.new.send(:perform, *args)
+          args = SidekiqBulkJob::Utils.load _args
+          if SidekiqBulkJob::Utils.class_with_method?(job_class_name)
+            job.send(method_name, *args)
+          else
+            job.new.send(method_name, *args)
+          end
         rescue Exception => e
-          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.full_message}")
+          error_handle.add _args, e
+          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.respond_to?(:full_message) ? e.full_message : e.message}")
           SidekiqBulkJob.fail_callback(job_class_name: job_class_name, args: args, exception: e)
-          SidekiqBulkJob::JobRetry.new(job, args, e).push
         end
       end
+      if error_handle.failed?
+        SidekiqBulkJob::JobRetry.new(job, error_handle).push
+      else
+        error_handle.clear
+      end
     end
   end
 end

data/lib/sidekiq_bulk_job/job_retry.rb

@@ -1,24 +1,26 @@
 require "sidekiq"
+require "sidekiq/job_retry"
 
 require "sidekiq_bulk_job/utils"
-require 'sidekiq/job_retry'
+require "sidekiq_bulk_job/bulk_error_handler"
 
 module SidekiqBulkJob
   class JobRetry
 
-    def initialize(klass, args, exception, options={})
+    def initialize(klass, error_handle, options={})
       @handler = Sidekiq::JobRetry.new(options)
       @klass = klass
-      @args = args
-      @exception = exception
+      @error_handle = error_handle
+      @retry_count = 0
     end
 
     def push(options={})
+      @retry_count = SidekiqBulkJob.redis.incr @error_handle.jid
       opts = job_options(options)
       queue_as = queue(@klass) || :default
       begin
         @handler.local(SidekiqBulkJob::BulkJob, opts, queue_as) do
-          raise @exception
+          raise @error_handle.raise_error
         end
       rescue Exception => e
       end
@@ -28,7 +30,12 @@ module SidekiqBulkJob
 
     def job_options(options={})
       # 0 retry: no retry and dead queue
-      opts = { 'class' => @klass.to_s, 'args' => @args, 'retry' => 0 }.merge(options)
+      opts = {
+        'class' => SidekiqBulkJob::BulkJob.to_s,
+        'args' => @error_handle.args,
+        'retry' => true,
+        'retry_count' => @retry_count.to_i
+      }.merge(options)
       if Sidekiq::VERSION >= "6.0.2"
         Sidekiq.dump_json(opts)
       else
@@ -37,8 +44,8 @@ module SidekiqBulkJob
     end
 
     def queue(woker)
-      if !woker.sidekiq_options.nil? && !woker.sidekiq_options.empty?
-        sidekiq_options = Utils.symbolize_keys(woker.sidekiq_options)
+      if woker.included_modules.include?(Sidekiq::Worker) && !woker.sidekiq_options.nil? && !woker.sidekiq_options.empty?
+        sidekiq_options = SidekiqBulkJob::Utils.symbolize_keys(woker.sidekiq_options)
         if !sidekiq_options[:queue].nil?
           sidekiq_options[:queue]
         end

data/lib/sidekiq_bulk_job/scheduled_job.rb

@@ -9,14 +9,19 @@ module SidekiqBulkJob
     sidekiq_options queue: :default, retry: false
 
     def perform(job_class_name, args_redis_key)
-      job = Utils.constantize(job_class_name)
+      target_name, method_name = SidekiqBulkJob::Utils.split_class_name_with_method job_class_name
+      job = SidekiqBulkJob::Utils.constantize(target_name)
       args_array = SidekiqBulkJob.flush args_redis_key
       args_array.each do |_args|
         begin
-          args = JSON.parse _args
-          job.new.send(:perform, *args)
+          args = SidekiqBulkJob::Utils.load _args
+          if SidekiqBulkJob::Utils.class_with_method?(job_class_name)
+            job.send(method_name, *args)
+          else
+            job.new.send(method_name, *args)
+          end
         rescue Exception => e
-          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.full_message}")
+          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.respond_to?(:full_message) ? e.full_message : e.message}")
           SidekiqBulkJob.fail_callback(job_class_name: job_class_name, args: args, exception: e)
           SidekiqBulkJob::JobRetry.new(job, args, e).push
         end

data/lib/sidekiq_bulk_job/utils.rb

@@ -1,72 +1,101 @@
-module Utils
+require 'yaml'
+require "sidekiq/extensions/active_record"
 
-  class << self
+module SidekiqBulkJob
+  module Utils
 
-    def symbolize_keys(obj)
-      case obj
-      when Array
-        obj.inject([]){|res, val|
-          res << case val
-          when Hash, Array
-            symbolize_keys(val)
-          else
-            val
-          end
-          res
-        }
-      when Hash
-        obj.inject({}){|res, (key, val)|
-          nkey = case key
-          when String
-            key.to_sym
-          else
-            key
-          end
-          nval = case val
-          when Hash, Array
-            symbolize_keys(val)
-          else
-            val
-          end
-          res[nkey] = nval
-          res
-        }
-      else
-        obj
+    class << self
+
+      def symbolize_keys(obj)
+        case obj
+        when Array
+          obj.inject([]){|res, val|
+            res << case val
+            when Hash, Array
+              symbolize_keys(val)
+            else
+              val
+            end
+            res
+          }
+        when Hash
+          obj.inject({}){|res, (key, val)|
+            nkey = case key
+            when String
+              key.to_sym
+            else
+              key
+            end
+            nval = case val
+            when Hash, Array
+              symbolize_keys(val)
+            else
+              val
+            end
+            res[nkey] = nval
+            res
+          }
+        else
+          obj
+        end
       end
-    end
 
-    def constantize(camel_cased_word)
-      names = camel_cased_word.split("::")
+      def constantize(camel_cased_word)
+        names = camel_cased_word.split("::")
 
-      # Trigger a built-in NameError exception including the ill-formed constant in the message.
-      Object.const_get(camel_cased_word) if names.empty?
+        # Trigger a built-in NameError exception including the ill-formed constant in the message.
+        Object.const_get(camel_cased_word) if names.empty?
 
-      # Remove the first blank element in case of '::ClassName' notation.
-      names.shift if names.size > 1 && names.first.empty?
+        # Remove the first blank element in case of '::ClassName' notation.
+        names.shift if names.size > 1 && names.first.empty?
 
-      names.inject(Object) do |constant, name|
-        if constant == Object
-          constant.const_get(name)
-        else
-          candidate = constant.const_get(name)
-          next candidate if constant.const_defined?(name, false)
-          next candidate unless Object.const_defined?(name)
+        names.inject(Object) do |constant, name|
+          if constant == Object
+            constant.const_get(name)
+          else
+            candidate = constant.const_get(name)
+            next candidate if constant.const_defined?(name, false)
+            next candidate unless Object.const_defined?(name)
+
+            # Go down the ancestors to check if it is owned directly. The check
+            # stops when we reach Object or the end of ancestors tree.
+            constant = constant.ancestors.inject(constant) do |const, ancestor|
+              break const    if ancestor == Object
+              break ancestor if ancestor.const_defined?(name, false)
+              const
+            end
 
-          # Go down the ancestors to check if it is owned directly. The check
-          # stops when we reach Object or the end of ancestors tree.
-          constant = constant.ancestors.inject(constant) do |const, ancestor|
-            break const    if ancestor == Object
-            break ancestor if ancestor.const_defined?(name, false)
-            const
+            # owner is in Object, so raise
+            constant.const_get(name, false)
           end
+        end
+      end
+
+      def class_with_method?(klass_name)
+        klass_name.include?('.')
+      end
+
+      def split_class_name_with_method(klass_name)
+        if class_with_method?(klass_name)
+          klass_name.split('.')
+        else
+          [klass_name, :perform]
+        end
+      end
+
+      def load yaml, legacy_filename = Object.new, filename: nil, fallback: false, symbolize_names: false
+        YAML.load yaml, legacy_filename, filename: filename, fallback: fallback, symbolize_names: symbolize_names
+      end
 
-          # owner is in Object, so raise
-          constant.const_get(name, false)
+      def dump o, io = nil, options = {}
+        marshalled = YAML.dump o, io, options
+        if marshalled.size > Sidekiq::Extensions::SIZE_LIMIT
+          SidekiqBulkJob.logger.warn { "job argument is #{marshalled.bytesize} bytes, you should refactor it to reduce the size" }
         end
+        marshalled
       end
+
     end
 
   end
-
 end

data/lib/sidekiq_bulk_job/version.rb

@@ -1,3 +1,3 @@
 module SidekiqBulkJob
-  VERSION = "0.1.1"
+  VERSION = "0.1.5"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq_bulk_job
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.5
 platform: ruby
 authors:
 - scalaview
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-11-09 00:00:00.000000000 Z
+date: 2021-08-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -73,6 +73,7 @@ files:
 - bin/setup
 - lib/sidekiq_bulk_job.rb
 - lib/sidekiq_bulk_job/batch_runner.rb
+- lib/sidekiq_bulk_job/bulk_error_handler.rb
 - lib/sidekiq_bulk_job/bulk_job.rb
 - lib/sidekiq_bulk_job/job_retry.rb
 - lib/sidekiq_bulk_job/monitor.rb
@@ -84,7 +85,7 @@ homepage: https://github.com/scalaview/sidekiq_bulk_job
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -99,8 +100,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.5.2
+signing_key:
 specification_version: 4
 summary: Collect same jobs to single worker, reduce job number and improve thread
   utilization.