sidekiq-paquet 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2a8cb3d7307887cc9c34999050e7460e6515f381
-  data.tar.gz: 6097a9b12ffc1546b784c2b519d5687da3017418
+  metadata.gz: 2a3886887de8e637e18bd9908831371f37ac7ea9
+  data.tar.gz: c424dee95bc830a67aad1b22581ea8b4924ea4a6
 SHA512:
-  metadata.gz: dc9e033a09c25ef09aae43065f9b92329acf33225c640d2737b4ef0dcc112681c5c436938206c0b67c4864d43ad8479a6533b78295ec05b8c1978cf60f216cc9
-  data.tar.gz: c970fde44941632e1d115dd88a5bbdfaf65e44f005ca13bcf356305d042283a705ca9ee1df46a5bd585de6a18b3fef92762a00c1b8ef5ab1b07e03fc0c9920c4
+  metadata.gz: 57cc001b4f3335cf0b53c712d62bd02f266b502589256b5456aa8d437ca952868f58f239c83deacca4e6b108a86789730feb5d5f0065ae3d50d6613025df3b48
+  data.tar.gz: 25d88e4e1ebcc4e956838e928575dfb325acbb1312a91309810278143c1d214c9128b285d5c128dab336a729115b6478d161471c86612055f00f662d65072e0d

data/.travis.yml

@@ -1,6 +1,8 @@
 language: ruby
+cache: bundler
 services:
   - redis-server
 rvm:
-  - 2.2.2
+  - 2.2.4
+  - 2.3.0
 before_install: gem install bundler -v 1.10.6

data/README.md

@@ -9,17 +9,17 @@ Useful for grouping background API calls or intensive database inserts coming fr
 gem install 'sidekiq-paquet'
 ```
 
-sidekiq-bulk requires Sidekiq 4+.
+sidekiq-paquet requires Sidekiq 4+.
 
 ## Usage
 
-Add `bulk: true` option to your worker's `sidekiq_options` to have jobs processed in bulk. The size of the bulk can be configured per worker. If not specified, the `Sidekiq::Paquet.options[:default_bulk_size]` is used.
+Add `bundled: true` option to your worker's `sidekiq_options` to have jobs processed in bulk. The size of the bundle can be configured per worker. If not specified, the `Sidekiq::Paquet.options[:default_bundle_size]` is used.
 
 ```ruby
 class ElasticIndexerWorker
   include Sidekiq::Worker
 
-  sidekiq_options bulk: true, bulk_size: 100
+  sidekiq_options bundled: true, bundle_size: 100
 
   def perform(*values)
     # Perform work with the array of values
@@ -27,7 +27,7 @@ class ElasticIndexerWorker
 end
 ```
 
-Instead of being processed by Sidekiq, jobs will be stored into a separate queue and periodically, a poller will retrieve them by slice of `bulk_size` and enqueue a regular Sidekiq job with that bulk as argument.
+Instead of being processed by Sidekiq right away, jobs will be stored into a separate queue and periodically, a separate thread will pick up this internal queue, slice `bundle_size` elements into an array and enqueue a regular Sidekiq job with that bundle as argument.
 Thus, your worker will only be invoked with an array of values, never with single values themselves.
 
 For example, if you call `perform_async` twice on the previous worker
@@ -41,23 +41,23 @@ the worker instance will receive these values as a single argument
 
 ```ruby
 [
-  { delete: { _index: 'users', _id: 1, _type: 'user' } },
-  { delete: { _index: 'users', _id: 2, _type: 'user' } }
+  [{ delete: { _index: 'users', _id: 1, _type: 'user' } }],
+  [{ delete: { _index: 'users', _id: 2, _type: 'user' } }]
 ]
 ```
 
-Every time polling happens, `sidekiq-paquet` will try to process all your workers marked for bulk. If you want to limit the time between two polling per worker, you can pass the `bulk_minimum_interval` option to sidekiq options.
+Every time flushing happens, `sidekiq-paquet` will try to process all your workers marked as bundled. If you want to limit the time between two flushing in a worker, you can pass the `minimum_execution_interval` option to sidekiq options.
 
 ## Configuration
 
 You can change global configuration by modifying the `Sidekiq::Paquet.options` hash.
 
 ```
-  Sidekiq::Paquet.options[:default_bulk_size] = 500 # Default is 100
-  Sidekiq::Paquet.options[:average_bulk_flush_interval] = 30 # Default is 15
+  Sidekiq::Paquet.options[:default_bundle_size] = 500 # Default is 100
+  Sidekiq::Paquet.options[:average_flush_interval] = 30 # Default is 15
 ```
 
-The `average_bulk_flush_interval` represent the average time elapsed between two polling of values. This scales with the number of sidekiq processes you're running. So if you have 5 sidekiq processes, and set the `average_bulk_flush_interval` to 15, each process will check for new bulk jobs every 75 seconds -- so that in average, the bulk queue will be checked every 15 seconds.
+The `average_flush_interval` represent the average time elapsed between two polling of values. This scales with the number of sidekiq processes you're running. So if you have 5 sidekiq processes, and set the `average_flush_interval` to 15, each process will check for new bundled jobs every 75 seconds -- so that in average, the bundles queue will be checked every 15 seconds.
 
 ## Contributing
 

data/lib/sidekiq/paquet/bundle.rb

@@ -0,0 +1,70 @@
+module Sidekiq
+  module Paquet
+    class Bundle
+
+      def self.append(item)
+        worker_name = item['class'.freeze]
+        args = item.fetch('args'.freeze, [])
+
+        Sidekiq.redis do |conn|
+          conn.multi do
+            conn.zadd('bundles'.freeze, 0, worker_name, nx: true)
+            conn.rpush("bundle:#{worker_name}", Sidekiq.dump_json(args))
+          end
+        end
+      end
+
+      def self.enqueue_jobs
+        now = Time.now.to_f
+        Sidekiq.redis do |conn|
+          workers = conn.zrangebyscore('bundles'.freeze, '-inf', now)
+
+          workers.each do |worker|
+            klass = worker.constantize
+            opts  = klass.get_sidekiq_options
+            min_interval = opts['minimum_execution_interval'.freeze]
+
+            items = conn.lrange("bundle:#{worker}", 0, -1)
+            items.map! { |i| Sidekiq.load_json(i) }
+
+            items.each_slice(opts.fetch('bundle_size'.freeze, Sidekiq::Paquet.options[:default_bundle_size])) do |vs|
+              Sidekiq::Client.push(
+                'class' => worker,
+                'queue' => opts['queue'.freeze],
+                'args'  => vs
+              )
+            end
+
+            conn.ltrim("bundle:#{worker}", items.size, -1)
+            conn.zadd('bundles'.freeze, now + min_interval, worker) if min_interval
+          end
+        end
+      end
+
+      def initialize(name)
+        @lname = "bundle:#{name}"
+      end
+
+      def queue
+        worker_name.constantize.get_sidekiq_options['queue'.freeze]
+      end
+
+      def worker_name
+        @lname.split(':').last
+      end
+
+      def size
+        Sidekiq.redis { |c| c.llen(@lname) }
+      end
+
+      def items
+        Sidekiq.redis { |c| c.lrange(@lname, 0, -1) }
+      end
+
+      def clear
+        Sidekiq.redis { |c| c.del(@lname) }
+      end
+
+    end
+  end
+end

data/lib/sidekiq/paquet/flusher.rb

@@ -0,0 +1,51 @@
+require 'concurrent/timer_task'
+
+module Sidekiq
+  module Paquet
+    class Flusher
+
+      def initialize
+        @task  = Concurrent::TimerTask.new(
+          execution_interval: execution_interval) { Bundle.enqueue_jobs }
+      end
+
+      def start
+        Sidekiq.logger.info('Starting paquet flusher')
+        @task.execute
+      end
+
+      def shutdown
+        Sidekiq.logger.info('Paquet flusher exiting...')
+        @task.shutdown
+      end
+
+      private
+
+      # To avoid having all processes flushing at the same time, randomize
+      # the execution interval between 0.5-1.5 the scaled interval, so that
+      # on average, interval is respected.
+      #
+      def execution_interval
+        avg = scaled_interval.to_f
+        avg * rand + avg / 2
+      end
+
+      # Scale interval with the number of Sidekiq processes running. Each one
+      # is going to run a flusher instance. If you have 10 processes and an
+      # average flush interval of 10s, it means one process is flushing every
+      # second, which is wasteful and beats the purpose of bundling.
+      #
+      # To avoid this, we scale the average flush interval with the number of
+      # Sidekiq processes running, i.e instead of flushing every 10s, let every
+      # process flush every 100 seconds.
+      #
+      def scaled_interval
+        Sidekiq::Paquet.options[:flush_interval] ||= begin
+          pcount = Sidekiq::ProcessSet.new.size
+          pcount = 1 if pcount == 0 # Maybe raise here
+          pcount * Sidekiq::Paquet.options[:average_flush_interval]
+        end
+      end
+    end
+  end
+end

data/lib/sidekiq/paquet/middleware.rb

@@ -2,8 +2,8 @@ module Sidekiq
   module Paquet
     class Middleware
       def call(worker, item, queue, redis_pool = nil)
-        if item['bulk'.freeze]
-          Batch.append(item)
+        if item['bundled'.freeze]
+          Bundle.append(item)
           false
         else
           yield

data/lib/sidekiq/paquet/version.rb

@@ -1,5 +1,5 @@
 module Sidekiq
   module Paquet
-    VERSION = '0.1.1'
+    VERSION = '0.2.0'
   end
 end

data/lib/sidekiq/paquet/views/index.erb

@@ -0,0 +1,21 @@
+<h3><%= t('Queues') %></h3>
+
+<div class="table_container">
+  <table class="table table-hover table-bordered table-striped table-white">
+    <thead>
+      <th><%= t('Worker') %></th>
+      <th><%= t('Queue') %></th>
+      <th><%= t('Size') %></th>
+      <th><%= t('Actions') %></th>
+    </thead>
+    <% @lists.each do |list| %>
+      <tr>
+        <td><%= list.worker_name %></td>
+        <td><%= list.queue %></td>
+        <td><%= number_with_delimiter(list.size) %> </td>
+        <td width="20%">
+        </td>
+      </tr>
+    <% end %>
+  </table>
+</div>

data/lib/sidekiq/paquet/web.rb

@@ -0,0 +1,20 @@
+require 'sidekiq/web'
+
+module Sidekiq
+  module Paquet
+    module Web
+      VIEWS = File.expand_path('views', File.dirname(__FILE__))
+
+      def self.registered(app)
+        app.get '/paquet' do
+          @lists = Sidekiq.redis { |c| c.zrange('bundles', 0, -1) }.map { |n| Bundle.new(n) }
+          erb File.read(File.join(VIEWS, 'index.erb'))
+        end
+      end
+
+    end
+  end
+end
+
+Sidekiq::Web.register(Sidekiq::Paquet::Web)
+Sidekiq::Web.tabs['Bundles'] = 'paquet'

data/lib/sidekiq/paquet.rb

@@ -1,18 +1,19 @@
+require 'concurrent/scheduled_task'
+
 require 'sidekiq'
 require 'sidekiq/paquet/version'
 
-require 'sidekiq/paquet/list'
-require 'sidekiq/paquet/batch'
+require 'sidekiq/paquet/bundle'
 require 'sidekiq/paquet/middleware'
-require 'sidekiq/paquet/poller'
+require 'sidekiq/paquet/flusher'
 
 module Sidekiq
   module Paquet
     DEFAULTS = {
-      default_bulk_size: 100,
-      bulk_flush_interval: nil,
-      average_bulk_flush_interval: 15,
-      dynamic_interval_scaling: true
+      default_bundle_size: 100,
+      flush_interval: nil,
+      average_flush_interval: 15,
+      initial_wait: 10
     }
 
     def self.options
@@ -22,6 +23,10 @@ module Sidekiq
     def self.options=(opts)
       @options = opts
     end
+
+    def self.initial_wait
+      options[:initial_wait] + (5 * rand)
+    end
   end
 end
 
@@ -37,11 +42,13 @@ Sidekiq.configure_server do |config|
   end
 
   config.on(:startup) do
-    config.options[:bulk_poller] = Sidekiq::Paquet::Poller.new
-    config.options[:bulk_poller].start
+    config.options[:paquet_flusher] = Sidekiq::Paquet::Flusher.new
+    Concurrent::ScheduledTask.execute(Sidekiq::Paquet.initial_wait) {
+      config.options[:paquet_flusher].start
+    }
   end
 
   config.on(:shutdown) do
-    config.options[:bulk_poller].terminate
+    config.options[:paquet_flusher].shutdown
   end
 end

data/{sidekiq-bulk.gemspec → sidekiq-paquet.gemspec}

@@ -19,8 +19,10 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_dependency "sidekiq", ">= 4"
+  spec.add_dependency "concurrent-ruby", "~> 1.0"
 
   spec.add_development_dependency "bundler", "~> 1.10"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "minitest"
+  spec.add_development_dependency "redis-namespace", "~> 1.5"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-paquet
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - ccocchi
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-03-18 00:00:00.000000000 Z
+date: 2016-03-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -24,6 +24,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '4'
+- !ruby/object:Gem::Dependency
+  name: concurrent-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -66,6 +80,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: redis-namespace
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
 description: 
 email:
 - cocchi.c@gmail.com
@@ -81,12 +109,13 @@ files:
 - README.md
 - Rakefile
 - lib/sidekiq/paquet.rb
-- lib/sidekiq/paquet/batch.rb
-- lib/sidekiq/paquet/list.rb
+- lib/sidekiq/paquet/bundle.rb
+- lib/sidekiq/paquet/flusher.rb
 - lib/sidekiq/paquet/middleware.rb
-- lib/sidekiq/paquet/poller.rb
 - lib/sidekiq/paquet/version.rb
-- sidekiq-bulk.gemspec
+- lib/sidekiq/paquet/views/index.erb
+- lib/sidekiq/paquet/web.rb
+- sidekiq-paquet.gemspec
 homepage: https://github.com/ccocchi/sidekiq-paquet
 licenses:
 - MIT

data/lib/sidekiq/paquet/batch.rb

@@ -1,46 +0,0 @@
-module Sidekiq
-  module Paquet
-    module Batch
-
-      def self.append(item)
-        worker_name = item['class'.freeze]
-        args = item.fetch('args'.freeze, [])
-
-        Sidekiq.redis do |conn|
-          conn.multi do
-            conn.zadd('bulks'.freeze, 0, worker_name, nx: true)
-            conn.rpush("bulk:#{worker_name}", Sidekiq.dump_json(args))
-          end
-        end
-      end
-
-      def self.enqueue_jobs
-        now = Time.now.to_f
-        Sidekiq.redis do |conn|
-          workers = conn.zrangebyscore('bulks'.freeze, '-inf', now)
-
-          workers.each do |worker|
-            klass = worker.constantize
-            opts  = klass.get_sidekiq_options
-            min_interval = opts['bulk_minimum_interval'.freeze]
-
-            items = conn.lrange("bulk:#{worker}", 0, -1)
-            items.map! { |i| Sidekiq.load_json(i) }
-
-            items.each_slice(opts.fetch('bulk_size'.freeze, Sidekiq::Paquet.options[:default_bulk_size])) do |vs|
-              Sidekiq::Client.push(
-                'class' => worker,
-                'queue' => opts['queue'.freeze],
-                'args'  => vs
-              )
-            end
-
-            conn.ltrim("bulk:#{worker}", items.size, -1)
-            conn.zadd('bulks'.freeze, now + min_interval, worker) if min_interval
-          end
-        end
-      end
-
-    end
-  end
-end

data/lib/sidekiq/paquet/list.rb

@@ -1,21 +0,0 @@
-module Sidekiq
-  module Paquet
-    class List
-      def initialize(name)
-        @lname = "bulk:#{name}"
-      end
-
-      def size
-        Sidekiq.redis { |c| c.llen(@lname) }
-      end
-
-      def items
-        Sidekiq.redis { |c| c.lrange(@lname, 0, -1) }
-      end
-
-      def clear
-        Sidekiq.redis { |c| c.del(@lname) }
-      end
-    end
-  end
-end

data/lib/sidekiq/paquet/poller.rb

@@ -1,84 +0,0 @@
-require 'sidekiq/util'
-require 'sidekiq/scheduled'
-
-module Sidekiq
-  module Paquet
-    class Poller < Sidekiq::Scheduled::Poller
-
-      def initialize
-        @sleeper  = ConnectionPool::TimedStack.new
-        @done     = false
-      end
-
-      def start
-        @thread ||= safe_thread('bulk') do
-          initial_wait
-
-          while !@done
-            enqueue
-            wait
-          end
-          Sidekiq.logger.info('Bulk exiting...')
-        end
-      end
-
-      def enqueue
-        begin
-          Batch.enqueue_jobs
-        rescue => ex
-          # Most likely a problem with redis networking.
-          # Punt and try again at the next interval
-          logger.error ex.message
-          logger.error ex.backtrace.first
-        end
-      end
-
-      private
-
-      # Calculates a random interval that is ±50% the desired average.
-      def random_poll_interval
-        avg = poll_interval_average.to_f
-        avg * rand + avg / 2
-      end
-
-      # We do our best to tune the poll interval to the size of the active Sidekiq
-      # cluster.  If you have 30 processes and poll every 15 seconds, that means one
-      # Sidekiq is checking Redis every 0.5 seconds - way too often for most people
-      # and really bad if the retry or scheduled sets are large.
-      #
-      # Instead try to avoid polling more than once every 15 seconds.  If you have
-      # 30 Sidekiq processes, we'll poll every 30 * 15 or 450 seconds.
-      # To keep things statistically random, we'll sleep a random amount between
-      # 225 and 675 seconds for each poll or 450 seconds on average.  Otherwise restarting
-      # all your Sidekiq processes at the same time will lead to them all polling at
-      # the same time: the thundering herd problem.
-      #
-      # We only do this if poll_interval_average is unset (the default).
-      def poll_interval_average
-        if Sidekiq::Paquet.options[:dynamic_interval_scaling]
-          scaled_poll_interval
-        else
-          Sidekiq::Paquet.options[:bulk_flush_interval] ||= scaled_poll_interval
-        end
-      end
-
-      # Calculates an average poll interval based on the number of known Sidekiq processes.
-      # This minimizes a single point of failure by dispersing check-ins but without taxing
-      # Redis if you run many Sidekiq processes.
-      def scaled_poll_interval
-        pcount = Sidekiq::ProcessSet.new.size
-        pcount = 1 if pcount == 0
-        pcount * Sidekiq::Paquet.options[:average_bulk_flush_interval]
-      end
-
-      def initial_wait
-        # Have all processes sleep between 5-15 seconds.  10 seconds
-        # to give time for the heartbeat to register (if the poll interval is going to be calculated by the number
-        # of workers), and 5 random seconds to ensure they don't all hit Redis at the same time.
-        total = INITIAL_WAIT + (15 * rand)
-        @sleeper.pop(total)
-      rescue Timeout::Error
-      end
-    end
-  end
-end