sidekiq_bulk_job 0.1.2 → 0.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: e3f6ce66f186f36e576e2611b892ecbb013f888b54f27bbc36f1a463aedc7b1f
-  data.tar.gz: 8deff846fe66b2fa565af200678aa20a67ba0ef1550ef920fb5930da0d76bd22
+SHA1:
+  metadata.gz: b24f02dbcf87a2fe41c7b73c64c92c698b8c6411
+  data.tar.gz: ddc7ec50e7055fe406c286309f599c07b4940390
 SHA512:
-  metadata.gz: 1199ea0291f826b28e7eb7e3fac6c5c00e98c69d37adc46e110e567ec82977ad3519c15fada8df89b07d5237161f457ab1c71271797897eff37f7677c0e26359
-  data.tar.gz: 1cb8a1f0c85caa3c05c994e1412b679251293d333b9d0b30831c13130e68a480035e484d73249e7be28645e01c7f237ea9dfd5348bdb1f97e16f73375faaa144
+  metadata.gz: 647ed0d297ed21e29729520a9f21a8dd6e7d54467b8ff7dcdac957f8cb4d48b0d879c71de33db9e51c152daa6d2a2ad24fa8a4491e7ee892b0e0f3eb791ac3d3
+  data.tar.gz: d401dd44b727304ca2e84acbc2a2d772eb8a81381e5f6ff0b609b72e3012ac9f12db4d708a953893ca03a84a60719ec7a7ff46766fe21f32bf74350d0d8f75f1

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    sidekiq_bulk_job (0.1.2)
+    sidekiq_bulk_job (0.1.5)
       sidekiq (~> 5.2.7)
 
 GEM
@@ -52,4 +52,4 @@ DEPENDENCIES
   sidekiq_bulk_job!
 
 BUNDLED WITH
-   2.1.4
+   2.2.25

data/README.md

@@ -1,8 +1,6 @@
 # SidekiqBulkJob
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/sidekiq_bulk_job`. To experiment with that code, run `bin/console` for an interactive prompt.
-
-TODO: Delete this and the text above, and describe your gem
+A tool to collect the same class jobs together and running in batch.
 
 ## Installation
 
@@ -22,26 +20,226 @@ Or install it yourself as:
 
 ## Usage
 
-###
+### Initialization：
+
+##### Parameters：
+
+* redis: redis client.
+* logger: log object，default Logger.new(STDOUT).
+* process_fail: a callback when the job fail.
+* async_delay: await delay time，default 60 seconds.
+* scheduled_delay: scheduled job delay time，default 10 seconds.
+* queue: default sidekiq running queue. By default the batch job will run at queue as same as sidekiq worker defined.
+* batch_size: batch size in same job，default 3000.
+* prefix: redis key prefix, default SidekiqBulkJob.
+
+```ruby
+	process_fail = lambda do |job_class_name, args, exception|
+	# do something
+	# send email
+	end
+	SidekiqBulkJob.config({
+		redis: Redis.new,
+		logger: Logger.new(STDOUT),
+		process_fail: process_fail,
+		async_delay: ASYNC_DELAY,
+		scheduled_delay: SCHEDULED_DELAY,
+		queue: :test,
+		batch_size: BATCH_SIZE,
+		prefix: "SidekiqBulkJob"
+	})
+	  // push a job
+	SidekiqBulkJob.perform_async(TestJob, 10)
+```
+
+### Usage
+
+At first define a TestJob as example
 ```ruby
-  process_fail = lambda do |job_class_name, args, exception|
-    # do somethine
-    # send email
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
   end
-  client = Redis.new
-  logger = Logger.new(STDOUT)
-  logger.level = Logger::WARN
-  SidekiqBulkJob.config redis: client, logger: logger, process_fail: process_fail, queue: :default, batch_size: 3000, prefix: "SidekiqBulkJob"
+  ```
+  
+##### Use SidekiqBulkJob async
 
-  // push a job
+  SidekiqBulkJob will collect the same job in to a list, a batch job will create when beyond the ```batch_size``` in  ```async_delay``` amount, and clear the list. The list will continue to collect the job which pushing inside. If reach the```async_delay``` time, the SidekiqBulkJob will also created to finish all job collected.
+  
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+
+  # simple use
   SidekiqBulkJob.perform_async(TestJob, 10)
+
+  # here will not create 1001 job in sidekiq
+  # now there are tow jobs created, one is collected 1000 TestJob in batch, another has 1 job inside.
+  (BATCH_SIZE + 1).times do |i|
+      SidekiqBulkJob.perform_async(TestJob, i)
+  end
 ```
 
-## Development
+##### Use SidekiqWork batch_perform_async to run async task
+
+```ruby
+# same as SidekiqBulkJob.perform_async(TestJob, 10)
+TestJob.batch_perform_async(10)
+```
+
+##### Use SidekiqBulkJob perform_at/perform_in to set scheduled task
+
+```ruby
+# run at 1 minute after with single job
+SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+# same as below
+SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+```
+
+##### Use SidekiqWork batch_perform_at/batch_perform_in to set scheduled task
+
+```ruby
+# same as SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+TestJob.batch_perform_at(1.minutes.after, 10)
+# same as SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+TestJob.batch_perform_in(1.minute, 10)
+```
+
+##### Use setter to set task
+
+```ruby
+# set queue to test and run async
+TestJob.set(queue: :test).batch_perform_async(10)
+# set queue to test and run after 90 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_async(10)
+
+#batch_perform_in first params interval will be overrided 'in'/'at' option at setter
+# run after 90 seconds instead of 10 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_in(10, 10)
+```
+
+## 中文
+
+### 初始化：
+
+##### 参数：
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+* redis: redis client
+* logger: 日志对象，默认Logger.new(STDOUT)
+* process_fail: 当job处理失败的通用回调
+* async_delay: 延迟等待时间，默认60秒
+* scheduled_delay: 定时任务延迟时间，默认10秒
+* queue: 默认运行队列。根据job本身设置的队列运行，当没有设置时候就使用这里设置的队列运行
+* batch_size: 同种类型job批量运行数量，默认3000
+* prefix: 存储到redis的前缀，默认SidekiqBulkJob
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+```ruby
+	process_fail = lambda do |job_class_name, args, exception|
+	# do something
+	# send email
+	end
+	SidekiqBulkJob.config({
+		redis: Redis.new,
+		logger: Logger.new(STDOUT),
+		process_fail: process_fail,
+		async_delay: ASYNC_DELAY,
+		scheduled_delay: SCHEDULED_DELAY,
+		queue: :test,
+		batch_size: BATCH_SIZE,
+		prefix: "SidekiqBulkJob"
+	})
+	  // push a job
+	SidekiqBulkJob.perform_async(TestJob, 10)
+```
+
+### 用法
+
+设置一个TestJob举例子
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+  ```
+  
+##### 使用SidekiqBulkJob的async接口
+
+SidekiqBulkJob会把同类型的job汇总到一个list中，当```async_delay```时间内超过```batch_size```，会新建一个batch job立刻执行汇总的全部jobs，清空list，清空的list会继续收集后续推入的job；如果在```async_delay```时间内未到达```batch_size```则会在最后一个job推入后等待```async_delay```时间创建一个batch job执行汇总的全部jobs
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+
+  # simple use
+  SidekiqBulkJob.perform_async(TestJob, 10)
+
+  # here will not create 1001 job in sidekiq
+  # now there are tow jobs created, one is collected 1000 TestJob in batch, another has 1 job inside.
+  (BATCH_SIZE + 1).times do |i|
+      SidekiqBulkJob.perform_async(TestJob, i)
+  end
+```
+
+##### 使用SidekiqWork的batch_perform_async接口异步执行任务
+
+```ruby
+# same as SidekiqBulkJob.perform_async(TestJob, 10)
+TestJob.batch_perform_async(10)
+```
+
+##### 使用SidekiqBulkJob的perform_at/perform_in接口设置定时任务
+
+```ruby
+# run at 1 minute after with single job
+SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+# same as below
+SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+```
+
+##### 使用SidekiqWork的batch_perform_at/batch_perform_in接口设置定时任务
+
+```ruby
+# same as SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+TestJob.batch_perform_at(1.minutes.after, 10)
+# same as SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+TestJob.batch_perform_in(1.minute, 10)
+```
+
+##### 使用setter设置
+
+```ruby
+# set queue to test and run async
+TestJob.set(queue: :test).batch_perform_async(10)
+# set queue to test and run after 90 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_async(10)
+
+#batch_perform_in first params interval will be overrided 'in'/'at' option at setter
+# run after 90 seconds instead of 10 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_in(10, 10)
+```
 
 ## Contributing
 

data/lib/sidekiq_bulk_job.rb

@@ -29,7 +29,7 @@ module SidekiqBulkJob
             job_class_name: job_class.to_s,
             perfrom_args: args,
             queue: options[:queue] || SidekiqBulkJob.queue
-          }.compact
+          }.select { |_, value| !value.nil? }
         SidekiqBulkJob.process payload
       else
         perform_in(options[:at] || options[:in], job_class, *args)
@@ -51,7 +51,7 @@ module SidekiqBulkJob
           at: ts,
           perfrom_args: args,
           queue: options[:queue] || SidekiqBulkJob.queue
-        }.compact
+        }.select { |_, value| !value.nil? }
         SidekiqBulkJob.process payload
       else
         perform_async(job_class, *args)
@@ -64,18 +64,37 @@ module SidekiqBulkJob
 
   class << self
 
-    attr_accessor :prefix, :redis, :queue, :batch_size, :logger, :process_fail
+    attr_accessor :prefix, :redis, :queue, :scheduled_delay, :async_delay, :batch_size, :logger, :process_fail
 
-    def config(redis: , logger: , process_fail: , queue: :default, batch_size: 3000, prefix: "SidekiqBulkJob")
+    def config(redis: , logger: , process_fail: , async_delay: 60, scheduled_delay: 10, queue: :default, batch_size: 3000, prefix: "SidekiqBulkJob")
       if redis.nil?
         raise ArgumentError.new("redis not allow nil")
       end
+      if logger.nil?
+        raise ArgumentError.new("logger not allow nil")
+      end
+      if process_fail.nil?
+        raise ArgumentError.new("process_fail not allow nil")
+      end
+      if async_delay.to_f < 2
+        raise ArgumentError.new("async_delay not allow less than 2 seconds.")
+      elsif async_delay.to_f > 5 * 60
+        raise ArgumentError.new("async_delay not allow greater than 5 minutes.")
+      end
+      if scheduled_delay.to_f < 2
+        raise ArgumentError.new("scheduled_delay not allow less than 2 seconds.")
+      elsif scheduled_delay.to_f > 30
+        raise ArgumentError.new("scheduled_delay not allow greater than 2 seconds.")
+      end
+
       self.redis = redis
       self.queue = queue
       self.batch_size = batch_size
       self.prefix = prefix
       self.logger = logger
       self.process_fail = process_fail
+      self.async_delay = async_delay.to_f
+      self.scheduled_delay = scheduled_delay.to_f
     end
 
     def set(options)
@@ -113,7 +132,7 @@ module SidekiqBulkJob
     def process(job_class_name: , at: nil, perfrom_args: [], queue: self.queue)
       if at.nil?
         key = generate_key(job_class_name)
-        client.lpush key, perfrom_args.to_json
+        client.lpush key, SidekiqBulkJob::Utils.dump(perfrom_args)
         bulk_run(job_class_name, key, queue: queue) if need_flush?(key)
         monitor(job_class_name, queue: queue)
       else
@@ -121,18 +140,18 @@ module SidekiqBulkJob
         args_redis_key = nil
         target = scheduled_set.find do |job|
           if job.klass == SidekiqBulkJob::ScheduledJob.to_s &&
-                job.at.to_i.between?((at - 5).to_i, (at + 30).to_i) # 允许30秒延迟
+                job.at.to_i.between?((at - self.scheduled_delay).to_i, (at + self.scheduled_delay).to_i) # 允许30秒延迟
               _job_class_name, args_redis_key = job.args
               _job_class_name == job_class_name
            end
         end
         if !target.nil? && !args_redis_key.nil? && !args_redis_key.empty?
           # 往现有的job参数set里增加参数
-          client.lpush args_redis_key, perfrom_args.to_json
+          client.lpush args_redis_key, SidekiqBulkJob::Utils.dump(perfrom_args)
         else
           # 新增加一个
           args_redis_key = SecureRandom.hex
-          client.lpush args_redis_key, perfrom_args.to_json
+          client.lpush args_redis_key, SidekiqBulkJob::Utils.dump(perfrom_args)
           SidekiqBulkJob::ScheduledJob.client_push("queue" => queue, "class" => SidekiqBulkJob::ScheduledJob, "at" => at, "args" => [job_class_name, args_redis_key])
         end
       end
@@ -197,7 +216,7 @@ module SidekiqBulkJob
       if !_monitor.nil?
         # TODO debug log
       else
-        SidekiqBulkJob::Monitor.client_push("queue" => queue, "at" => (time_now + 60).to_f, "class" => SidekiqBulkJob::Monitor, "args" => [time_now.to_f, job_class_name])
+        SidekiqBulkJob::Monitor.client_push("queue" => queue, "at" => (time_now + self.async_delay).to_f, "class" => SidekiqBulkJob::Monitor, "args" => [time_now.to_f, job_class_name])
       end
     end
 

data/lib/sidekiq_bulk_job/bulk_error_handler.rb

@@ -0,0 +1,59 @@
+module SidekiqBulkJob
+  class BulkErrorHandler
+
+    ErrorCollection = Struct.new(:args, :exception) do
+      def message
+        exception.message
+      end
+
+      def backtrace
+        exception.backtrace
+      end
+    end
+
+    attr_accessor :job_class_name, :errors, :jid
+
+    def initialize(job_class_name, jid)
+      @jid = jid
+      @job_class_name = job_class_name
+      @errors = []
+    end
+
+    def add(job_args, exception)
+      errors << ErrorCollection.new(job_args, exception)
+    end
+
+    def backtrace
+      errors.map(&:backtrace).flatten
+    end
+
+    def args
+      [job_class_name, errors.map(&:args)]
+    end
+
+    def failed?
+      !errors.empty?
+    end
+
+    def raise_error
+      error = BulkError.new(errors.map(&:message).join('; '))
+      error.set_backtrace self.backtrace
+      error
+    end
+
+    def retry_count
+      SidekiqBulkJob.redis.incr jid
+    end
+
+    def clear
+      SidekiqBulkJob.redis.del jid
+    end
+
+    class BulkError < StandardError
+      def initialize(message)
+        super(message)
+      end
+    end
+
+  end
+end

data/lib/sidekiq_bulk_job/bulk_job.rb

@@ -1,30 +1,37 @@
 require "sidekiq"
 
 require "sidekiq_bulk_job/job_retry"
+require "sidekiq_bulk_job/bulk_error_handler"
 require "sidekiq_bulk_job/utils"
 
 module SidekiqBulkJob
   class BulkJob
     include Sidekiq::Worker
-    sidekiq_options queue: :default, retry: false
+    sidekiq_options queue: :default, retry: true
 
     def perform(job_class_name, args_array)
       target_name, method_name = SidekiqBulkJob::Utils.split_class_name_with_method job_class_name
       job = SidekiqBulkJob::Utils.constantize(target_name)
+      error_handle = BulkErrorHandler.new(job_class_name, self.jid)
       args_array.each do |_args|
         begin
-          args = JSON.parse _args
+          args = SidekiqBulkJob::Utils.load _args
           if SidekiqBulkJob::Utils.class_with_method?(job_class_name)
             job.send(method_name, *args)
           else
             job.new.send(method_name, *args)
           end
         rescue Exception => e
-          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.full_message}")
+          error_handle.add _args, e
+          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.respond_to?(:full_message) ? e.full_message : e.message}")
           SidekiqBulkJob.fail_callback(job_class_name: job_class_name, args: args, exception: e)
-          SidekiqBulkJob::JobRetry.new(job, args, e).push
         end
       end
+      if error_handle.failed?
+        SidekiqBulkJob::JobRetry.new(job, error_handle).push
+      else
+        error_handle.clear
+      end
     end
   end
 end

data/lib/sidekiq_bulk_job/job_retry.rb

@@ -1,24 +1,26 @@
 require "sidekiq"
+require "sidekiq/job_retry"
 
 require "sidekiq_bulk_job/utils"
-require 'sidekiq/job_retry'
+require "sidekiq_bulk_job/bulk_error_handler"
 
 module SidekiqBulkJob
   class JobRetry
 
-    def initialize(klass, args, exception, options={})
+    def initialize(klass, error_handle, options={})
       @handler = Sidekiq::JobRetry.new(options)
       @klass = klass
-      @args = args
-      @exception = exception
+      @error_handle = error_handle
+      @retry_count = 0
     end
 
     def push(options={})
+      @retry_count = SidekiqBulkJob.redis.incr @error_handle.jid
       opts = job_options(options)
       queue_as = queue(@klass) || :default
       begin
         @handler.local(SidekiqBulkJob::BulkJob, opts, queue_as) do
-          raise @exception
+          raise @error_handle.raise_error
         end
       rescue Exception => e
       end
@@ -28,7 +30,12 @@ module SidekiqBulkJob
 
     def job_options(options={})
       # 0 retry: no retry and dead queue
-      opts = { 'class' => @klass.to_s, 'args' => @args, 'retry' => 0 }.merge(options)
+      opts = {
+        'class' => SidekiqBulkJob::BulkJob.to_s,
+        'args' => @error_handle.args,
+        'retry' => true,
+        'retry_count' => @retry_count.to_i
+      }.merge(options)
       if Sidekiq::VERSION >= "6.0.2"
         Sidekiq.dump_json(opts)
       else

data/lib/sidekiq_bulk_job/scheduled_job.rb

@@ -14,14 +14,14 @@ module SidekiqBulkJob
       args_array = SidekiqBulkJob.flush args_redis_key
       args_array.each do |_args|
         begin
-          args = JSON.parse _args
+          args = SidekiqBulkJob::Utils.load _args
           if SidekiqBulkJob::Utils.class_with_method?(job_class_name)
             job.send(method_name, *args)
           else
             job.new.send(method_name, *args)
           end
         rescue Exception => e
-          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.full_message}")
+          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.respond_to?(:full_message) ? e.full_message : e.message}")
           SidekiqBulkJob.fail_callback(job_class_name: job_class_name, args: args, exception: e)
           SidekiqBulkJob::JobRetry.new(job, args, e).push
         end

data/lib/sidekiq_bulk_job/utils.rb

@@ -1,3 +1,6 @@
+require 'yaml'
+require "sidekiq/extensions/active_record"
+
 module SidekiqBulkJob
   module Utils
 
@@ -80,6 +83,18 @@ module SidekiqBulkJob
         end
       end
 
+      def load yaml, legacy_filename = Object.new, filename: nil, fallback: false, symbolize_names: false
+        YAML.load yaml, legacy_filename, filename: filename, fallback: fallback, symbolize_names: symbolize_names
+      end
+
+      def dump o, io = nil, options = {}
+        marshalled = YAML.dump o, io, options
+        if marshalled.size > Sidekiq::Extensions::SIZE_LIMIT
+          SidekiqBulkJob.logger.warn { "job argument is #{marshalled.bytesize} bytes, you should refactor it to reduce the size" }
+        end
+        marshalled
+      end
+
     end
 
   end

data/lib/sidekiq_bulk_job/version.rb

@@ -1,3 +1,3 @@
 module SidekiqBulkJob
-  VERSION = "0.1.2"
+  VERSION = "0.1.6"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq_bulk_job
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.6
 platform: ruby
 authors:
 - scalaview
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-11-11 00:00:00.000000000 Z
+date: 2021-08-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -73,6 +73,7 @@ files:
 - bin/setup
 - lib/sidekiq_bulk_job.rb
 - lib/sidekiq_bulk_job/batch_runner.rb
+- lib/sidekiq_bulk_job/bulk_error_handler.rb
 - lib/sidekiq_bulk_job/bulk_job.rb
 - lib/sidekiq_bulk_job/job_retry.rb
 - lib/sidekiq_bulk_job/monitor.rb
@@ -84,7 +85,7 @@ homepage: https://github.com/scalaview/sidekiq_bulk_job
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -99,8 +100,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.5.2
+signing_key:
 specification_version: 4
 summary: Collect same jobs to single worker, reduce job number and improve thread
   utilization.