sidekiq_bulk_job 0.1.4 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: 795d7a4f3d359d67ee4cf6894df0fc14193da298e778e779da208f38168f6a96
-  data.tar.gz: baf26c1a2d731da06faed59c9824298e4c22c7ae3cf8a67e4fb7c29fa2d4860e
+SHA1:
+  metadata.gz: ca8157b27f077c0db69ac593ad1fcbd8c14e0eab
+  data.tar.gz: 6c87da78b2105230eb522ff3dc6c09f4c73c3540
 SHA512:
-  metadata.gz: ac4f23ee0e7a81ad91ee343c3c5df63baf656d48a3eebe2925f18cd6e87589f519c477f3f2c49c4cf0fa923586506591ee81014fce20a8ff848ab63a8df0b195
-  data.tar.gz: b770c30598a61234cbcd1534c84ed9f7f0782b47f3315c51e1aa2368b2a82eabdcc1cdbc8b062bbce491e7b726ef662e74448dd6474abd9a89be5c9d6ad7cace
+  metadata.gz: 5ba3674183b7f3fa818d4c2bec3f8b6d3f8a632afcd98d38a3beff78e165ad8fe2078d0149dc7601e6e78ec80914f10cba35f819707f42f45cef756fe39cf546
+  data.tar.gz: afe1b3f3787cfe828da123984323cb0b45d4877388238b7b080e8b5ea33d1e718e5225ac6f32e8467a2f77dc7340fd791ad17a3e22e8166d3c9a0c03bf8ca3b7

data/README.md

@@ -1,8 +1,6 @@
 # SidekiqBulkJob
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/sidekiq_bulk_job`. To experiment with that code, run `bin/console` for an interactive prompt.
-
-TODO: Delete this and the text above, and describe your gem
+A tool to collect the same class jobs together and running in batch.
 
 ## Installation
 
@@ -22,26 +20,226 @@ Or install it yourself as:
 
 ## Usage
 
-###
+### Initialization：
+
+##### Parameters：
+
+* redis: redis client.
+* logger: log object，default Logger.new(STDOUT).
+* process_fail: a callback when the job fail.
+* async_delay: await delay time，default 60 seconds.
+* scheduled_delay: scheduled job delay time，default 10 seconds.
+* queue: default sidekiq running queue. By default the batch job will run at queue as same as sidekiq worker defined.
+* batch_size: batch size in same job，default 3000.
+* prefix: redis key prefix, default SidekiqBulkJob.
+
+```ruby
+	process_fail = lambda do |job_class_name, args, exception|
+	# do something
+	# send email
+	end
+	SidekiqBulkJob.config({
+		redis: Redis.new,
+		logger: Logger.new(STDOUT),
+		process_fail: process_fail,
+		async_delay: ASYNC_DELAY,
+		scheduled_delay: SCHEDULED_DELAY,
+		queue: :test,
+		batch_size: BATCH_SIZE,
+		prefix: "SidekiqBulkJob"
+	})
+	  // push a job
+	SidekiqBulkJob.perform_async(TestJob, 10)
+```
+
+### Usage
+
+At first define a TestJob as example
 ```ruby
-  process_fail = lambda do |job_class_name, args, exception|
-    # do somethine
-    # send email
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
   end
-  client = Redis.new
-  logger = Logger.new(STDOUT)
-  logger.level = Logger::WARN
-  SidekiqBulkJob.config redis: client, logger: logger, process_fail: process_fail, queue: :default, batch_size: 3000, prefix: "SidekiqBulkJob"
+  ```
+  
+##### Use SidekiqBulkJob async
 
-  // push a job
+  SidekiqBulkJob will collect the same job in to a list, a batch job will create when beyond the ```batch_size``` in  ```async_delay``` amount, and clear the list. The list will continue to collect the job which pushing inside. If reach the```async_delay``` time, the SidekiqBulkJob will also created to finish all job collected.
+  
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+
+  # simple use
   SidekiqBulkJob.perform_async(TestJob, 10)
+
+  # here will not create 1001 job in sidekiq
+  # now there are tow jobs created, one is collected 1000 TestJob in batch, another has 1 job inside.
+  (BATCH_SIZE + 1).times do |i|
+      SidekiqBulkJob.perform_async(TestJob, i)
+  end
 ```
 
-## Development
+##### Use SidekiqWork batch_perform_async to run async task
+
+```ruby
+# same as SidekiqBulkJob.perform_async(TestJob, 10)
+TestJob.batch_perform_async(10)
+```
+
+##### Use SidekiqBulkJob perform_at/perform_in to set scheduled task
+
+```ruby
+# run at 1 minute after with single job
+SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+# same as below
+SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+```
+
+##### Use SidekiqWork batch_perform_at/batch_perform_in to set scheduled task
+
+```ruby
+# same as SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+TestJob.batch_perform_at(1.minutes.after, 10)
+# same as SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+TestJob.batch_perform_in(1.minute, 10)
+```
+
+##### Use setter to set task
+
+```ruby
+# set queue to test and run async
+TestJob.set(queue: :test).batch_perform_async(10)
+# set queue to test and run after 90 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_async(10)
+
+#batch_perform_in first params interval will be overrided 'in'/'at' option at setter
+# run after 90 seconds instead of 10 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_in(10, 10)
+```
+
+## 中文
+
+### 初始化：
+
+##### 参数：
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+* redis: redis client
+* logger: 日志对象，默认Logger.new(STDOUT)
+* process_fail: 当job处理失败的通用回调
+* async_delay: 延迟等待时间，默认60秒
+* scheduled_delay: 定时任务延迟时间，默认10秒
+* queue: 默认运行队列。根据job本身设置的队列运行，当没有设置时候就使用这里设置的队列运行
+* batch_size: 同种类型job批量运行数量，默认3000
+* prefix: 存储到redis的前缀，默认SidekiqBulkJob
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+```ruby
+	process_fail = lambda do |job_class_name, args, exception|
+	# do something
+	# send email
+	end
+	SidekiqBulkJob.config({
+		redis: Redis.new,
+		logger: Logger.new(STDOUT),
+		process_fail: process_fail,
+		async_delay: ASYNC_DELAY,
+		scheduled_delay: SCHEDULED_DELAY,
+		queue: :test,
+		batch_size: BATCH_SIZE,
+		prefix: "SidekiqBulkJob"
+	})
+	  // push a job
+	SidekiqBulkJob.perform_async(TestJob, 10)
+```
+
+### 用法
+
+设置一个TestJob举例子
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+  ```
+  
+##### 使用SidekiqBulkJob的async接口
+
+SidekiqBulkJob会把同类型的job汇总到一个list中，当```async_delay```时间内超过```batch_size```，会新建一个batch job立刻执行汇总的全部jobs，清空list，清空的list会继续收集后续推入的job；如果在```async_delay```时间内未到达```batch_size```则会在最后一个job推入后等待```async_delay```时间创建一个batch job执行汇总的全部jobs
+```ruby
+# create a sidekiq worker, use default queue
+  class TestJob
+    include Sidekiq::Worker
+    sidekiq_options queue: :default
+
+    def perform(*args)
+      puts args
+    end
+  end
+
+  # simple use
+  SidekiqBulkJob.perform_async(TestJob, 10)
+
+  # here will not create 1001 job in sidekiq
+  # now there are tow jobs created, one is collected 1000 TestJob in batch, another has 1 job inside.
+  (BATCH_SIZE + 1).times do |i|
+      SidekiqBulkJob.perform_async(TestJob, i)
+  end
+```
+
+##### 使用SidekiqWork的batch_perform_async接口异步执行任务
+
+```ruby
+# same as SidekiqBulkJob.perform_async(TestJob, 10)
+TestJob.batch_perform_async(10)
+```
+
+##### 使用SidekiqBulkJob的perform_at/perform_in接口设置定时任务
+
+```ruby
+# run at 1 minute after with single job
+SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+# same as below
+SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+```
+
+##### 使用SidekiqWork的batch_perform_at/batch_perform_in接口设置定时任务
+
+```ruby
+# same as SidekiqBulkJob.perform_at(1.minutes.after, TestJob, 10)
+TestJob.batch_perform_at(1.minutes.after, 10)
+# same as SidekiqBulkJob.perform_in(1 * 60, TestJob, 10)
+TestJob.batch_perform_in(1.minute, 10)
+```
+
+##### 使用setter设置
+
+```ruby
+# set queue to test and run async
+TestJob.set(queue: :test).batch_perform_async(10)
+# set queue to test and run after 90 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_async(10)
+
+#batch_perform_in first params interval will be overrided 'in'/'at' option at setter
+# run after 90 seconds instead of 10 seconds
+TestJob.set(queue: :test, in: 90).batch_perform_in(10, 10)
+```
 
 ## Contributing
 

data/lib/sidekiq_bulk_job.rb

@@ -29,7 +29,7 @@ module SidekiqBulkJob
             job_class_name: job_class.to_s,
             perfrom_args: args,
             queue: options[:queue] || SidekiqBulkJob.queue
-          }.compact
+          }.select { |_, value| !value.nil? }
         SidekiqBulkJob.process payload
       else
         perform_in(options[:at] || options[:in], job_class, *args)
@@ -51,7 +51,7 @@ module SidekiqBulkJob
           at: ts,
           perfrom_args: args,
           queue: options[:queue] || SidekiqBulkJob.queue
-        }.compact
+        }.select { |_, value| !value.nil? }
         SidekiqBulkJob.process payload
       else
         perform_async(job_class, *args)

data/lib/sidekiq_bulk_job/bulk_error_handler.rb

@@ -0,0 +1,59 @@
+module SidekiqBulkJob
+  class BulkErrorHandler
+
+    ErrorCollection = Struct.new(:args, :exception) do
+      def message
+        exception.message
+      end
+
+      def backtrace
+        exception.backtrace
+      end
+    end
+
+    attr_accessor :job_class_name, :errors, :jid
+
+    def initialize(job_class_name, jid)
+      @jid = jid
+      @job_class_name = job_class_name
+      @errors = []
+    end
+
+    def add(job_args, exception)
+      errors << ErrorCollection.new(job_args, exception)
+    end
+
+    def backtrace
+      errors.map(&:backtrace).flatten
+    end
+
+    def args
+      errors.map(&:args)
+    end
+
+    def failed?
+      !errors.empty?
+    end
+
+    def raise_error
+      error = BulkError.new(errors.map(&:message).join('; '))
+      error.set_backtrace self.backtrace
+      error
+    end
+
+    def retry_count
+      SidekiqBulkJob.redis.incr jid
+    end
+
+    def clear
+      SidekiqBulkJob.redis.del jid
+    end
+
+    class BulkError < StandardError
+      def initialize(message)
+        super(message)
+      end
+    end
+
+  end
+end

data/lib/sidekiq_bulk_job/bulk_job.rb

@@ -1,16 +1,18 @@
 require "sidekiq"
 
 require "sidekiq_bulk_job/job_retry"
+require "sidekiq_bulk_job/bulk_error_handler"
 require "sidekiq_bulk_job/utils"
 
 module SidekiqBulkJob
   class BulkJob
     include Sidekiq::Worker
-    sidekiq_options queue: :default, retry: false
+    sidekiq_options queue: :default, retry: true
 
     def perform(job_class_name, args_array)
       target_name, method_name = SidekiqBulkJob::Utils.split_class_name_with_method job_class_name
       job = SidekiqBulkJob::Utils.constantize(target_name)
+      error_handle = BulkErrorHandler.new(job_class_name, self.jid)
       args_array.each do |_args|
         begin
           args = SidekiqBulkJob::Utils.load _args
@@ -20,11 +22,16 @@ module SidekiqBulkJob
             job.new.send(method_name, *args)
           end
         rescue Exception => e
-          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.full_message}")
+          error_handle.add _args, e
+          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.respond_to?(:full_message) ? e.full_message : e.message}")
           SidekiqBulkJob.fail_callback(job_class_name: job_class_name, args: args, exception: e)
-          SidekiqBulkJob::JobRetry.new(job, args, e).push
         end
       end
+      if error_handle.failed?
+        SidekiqBulkJob::JobRetry.new(job, error_handle).push
+      else
+        error_handle.clear
+      end
     end
   end
 end

data/lib/sidekiq_bulk_job/job_retry.rb

@@ -1,24 +1,26 @@
 require "sidekiq"
+require "sidekiq/job_retry"
 
 require "sidekiq_bulk_job/utils"
-require 'sidekiq/job_retry'
+require "sidekiq_bulk_job/bulk_error_handler"
 
 module SidekiqBulkJob
   class JobRetry
 
-    def initialize(klass, args, exception, options={})
+    def initialize(klass, error_handle, options={})
       @handler = Sidekiq::JobRetry.new(options)
       @klass = klass
-      @args = args
-      @exception = exception
+      @error_handle = error_handle
+      @retry_count = 0
     end
 
     def push(options={})
+      @retry_count = SidekiqBulkJob.redis.incr @error_handle.jid
       opts = job_options(options)
       queue_as = queue(@klass) || :default
       begin
         @handler.local(SidekiqBulkJob::BulkJob, opts, queue_as) do
-          raise @exception
+          raise @error_handle.raise_error
         end
       rescue Exception => e
       end
@@ -28,7 +30,12 @@ module SidekiqBulkJob
 
     def job_options(options={})
       # 0 retry: no retry and dead queue
-      opts = { 'class' => @klass.to_s, 'args' => @args, 'retry' => 0 }.merge(options)
+      opts = {
+        'class' => SidekiqBulkJob::BulkJob.to_s,
+        'args' => @error_handle.args,
+        'retry' => true,
+        'retry_count' => @retry_count.to_i
+      }.merge(options)
       if Sidekiq::VERSION >= "6.0.2"
         Sidekiq.dump_json(opts)
       else

data/lib/sidekiq_bulk_job/scheduled_job.rb

@@ -21,7 +21,7 @@ module SidekiqBulkJob
             job.new.send(method_name, *args)
           end
         rescue Exception => e
-          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.full_message}")
+          SidekiqBulkJob.logger.error("#{job_class_name} Args: #{args}, Error: #{e.respond_to?(:full_message) ? e.full_message : e.message}")
           SidekiqBulkJob.fail_callback(job_class_name: job_class_name, args: args, exception: e)
           SidekiqBulkJob::JobRetry.new(job, args, e).push
         end

data/lib/sidekiq_bulk_job/version.rb

@@ -1,3 +1,3 @@
 module SidekiqBulkJob
-  VERSION = "0.1.4"
+  VERSION = "0.1.5"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq_bulk_job
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.5
 platform: ruby
 authors:
 - scalaview
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-11-16 00:00:00.000000000 Z
+date: 2021-08-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -73,6 +73,7 @@ files:
 - bin/setup
 - lib/sidekiq_bulk_job.rb
 - lib/sidekiq_bulk_job/batch_runner.rb
+- lib/sidekiq_bulk_job/bulk_error_handler.rb
 - lib/sidekiq_bulk_job/bulk_job.rb
 - lib/sidekiq_bulk_job/job_retry.rb
 - lib/sidekiq_bulk_job/monitor.rb
@@ -84,7 +85,7 @@ homepage: https://github.com/scalaview/sidekiq_bulk_job
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -99,8 +100,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.5.2
+signing_key:
 specification_version: 4
 summary: Collect same jobs to single worker, reduce job number and improve thread
   utilization.