nexia_worker_roulette 0.1.11

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4ea8533bab4bfc550f23023bfa81f0befbcd107d
+  data.tar.gz: 0daa96c1b20b1c1621bbb4098b5726ff645e283a
+SHA512:
+  metadata.gz: 507641dc68cffaaccdb6c5077e38039af0afce8b348f2c240a60048b76b04f3472a5246b04bfd671c6b3a41b5ee080e469844cc95020e6cc8f1457f8faca1d4f
+  data.tar.gz: 42b459a4d9288efd387c5c15d56c490c857de003d09d16d1f2aeec6bd3f2d39e4afa9ffd12a9faddd7454ae46c78d798d273ffc1ff93455fdb1e78568b4d722d

data/.agignore

@@ -0,0 +1,2 @@
+coverage/*
+vendor/bundle/*

data/.gitignore

@@ -0,0 +1,21 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+tmp/*
+coverage/*
+vendor/bundle
+results.xml

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format doc

data/.simplecov

@@ -0,0 +1,16 @@
+require 'simplecov'
+require 'simplecov-rcov'
+
+class SimpleCov::Formatter::MergedFormatter
+  def format(result)
+     SimpleCov::Formatter::HTMLFormatter.new.format(result)
+     SimpleCov::Formatter::RcovFormatter.new.format(result)
+  end
+end
+SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter
+
+SimpleCov.start do
+  add_filter "_spec.rb"
+
+  SimpleCov.minimum_coverage 99
+end

data/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in worker_roulette.gemspec
+gemspec
+
+# Gemfile
+group :development do
+  gem 'evented-spec', :git => 'https://github.com/ruby-amqp/evented-spec.git'
+end

data/Guardfile

@@ -0,0 +1,3 @@
+guard 'rspec' do  
+  watch(%r{.*})
+end

data/README.md

@@ -0,0 +1,114 @@
+# WorkerRoulette
+
+WorkerRoulette is designed to allow large numbers of unique devices, processes, users, or whatever communicate over individual channels without messing up the order of their messages. WorkerRoulette was created to solve two otherwise hard problems. First, other messaging solutions (I'm looking at you RabbitMQ) are not designed to  handle very large numbers of queues (millions); because WorkerRoulette is built on top of Redis, we have successfully tested it running with millions of queues. Second, other messaging systems assume one (or more) of three things: 1. Your message consumers know the routing key of messages they are interested in processing; 2. Your messages can wait so that only one consumer is processed at a time; 3. You love to complicated write code to put your messages back in order. Sometimes, none of these things is true and that is where WorkerRoulette comes in.
+
+WorkerRoulette lets you have thousands of competing consumers (distributed over as many machines as you'd like) processing ordered messages from millions of totally unknown message providers. It does all this and ensures that the messages sent from each message provider are processed in exactly the order it sent them.
+
+## Use
+```ruby
+size_of_connection_pool = 100
+
+#Start it up
+#the config takes size for the connection pool size, evented to specify whether to use evented redis or not,
+#polling_time dictates the number of seconds to wait before checking the job queue
+#(use higher numbers for high traffic systems), then the normal redis config
+WorkerRoulette.start(size: size_of_connection_pool, evented: false, host: 'localhost', timeout: 5, db: 1, polling_time: 2)
+
+#Enqueue some work
+sender_id = :shady
+foreman = WorkerRoulette.foreman(sender_id)
+foreman.enqueue_work_order('hello')
+
+#Pull it off (headers always include both the unique id of the sender)
+tradesman = WorkerRoulette.tradesman
+work_orders = tradesman.work_orders! #drain the queue of the next available sender
+work_orders # => {'headers' => {'sender' => :shady}, 'payload' => ['hello']}
+
+#Enqueue some more from someone else
+other_sender_id = :the_real_slim_shady
+other_foreman = WorkerRoulette.foreman(other_sender_id, 'some_namespace')
+other_foreman.enqueue_work_order({'can you get me' => 'the number nine?'}, {'custom' => 'headers'})
+
+#Have the same worker pull that off
+work_orders = tradesman.work_orders! #drain the queue of the next available sender
+work_orders # => {'headers' => {'sender' => :the_real_slim_shady, 'custom' => 'headers'},
+            #     'payload' => [{'can you get me' => 'the number nine?'}]}
+
+#Have your workers wait for work to come in
+foreman.enqueue_work_order('will I see you later?')
+foreman.enqueue_work_order('can you give me back my dime?')
+
+#And they will pull it off as it comes, as long as it comes. If the job board is empty (ie there is now work to do),
+#this method will poll redis every "polling_time" seconds until it finds work,
+#at which point it will continue processing jobs until the job board is empty again
+tradesman.wait_for_work_orders do |work_orders| #drain the queue of the next available sender
+  work_orders.first # => ['will I see you later', 'can you give me back my dime?']
+end
+```
+
+## Channels
+You can also namespace your work orders over a channel, in case you have several sorts of competing consumers who should not step on each other's toes:
+```ruby
+tradesman         = WorkerRoulette.tradesman('good_channel')
+tradesman.should_receive(:work_orders!).and_call_original
+
+good_foreman      = WorkerRoulette.foreman('foreman', 'good_channel')
+bad_foreman       = WorkerRoulette.foreman('foreman', 'bad_channel')
+
+good_foreman.enqueue_work_order('some old fashion work')
+bad_foreman.enqueue_work_order('evil biddings you should not carry out')
+
+tradesman.wait_for_work_orders do |work|
+  work.to_s.should match("some old fashion work") #only got the work from the good foreman
+  work.to_s.should_not match("evil")              #channels let us ignore the other's evil orders
+end
+```
+
+## Performance
+Running the performance tests on my laptop, the numbers break down like this:
+### Evented
+  - Polling: ~11,500 read-write round-trips / second
+  - Manual:  ~6,000  read-write round-trips / second
+
+### Non-Evented
+  - Polling: ~10,000 read-write round-trips / second
+  - Manual:  ~5,500  read-write round-trips / second
+
+To run the perf tests yourself run `bundle exec spec:perf`
+
+## Redis Version
+WorkerRoulette uses Redis' lua scripting feature to acheive such high throughput and therefore requires a version of Redis that supports lua scripting (>= Redis 2.6)
+
+##Caveat Emptor
+While WorkerRoulette does promise to keep the messages of each consumer processed in order by competing consumers, it does NOT guarantee the order in which the queues themselves will be processed. In general, work is processed in a FIFO order, but for performance reasons this has been left a loose FIFO. For example, if Abdul enqueues some ordered messages ('1', '2', and '3') and then so do Mark and Wanda, Mark's messages may be processed first, then it would likely be Abdul's, and then Wanda's. However, even though Mark jumped the line, Abdul's messages will still be processed in the order he enqueued them ('1', '2', then '3').
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'worker_roulette'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install worker_roulette
+
+## Run the specs
+
+    $ bundle exec rake spec:ci
+
+## Run the performance tests
+
+    $ bundle exec rake spec:perf
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,32 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new
+
+task :default => :spec
+task :test => :spec
+
+def rspec_out_file
+  require 'rspec_junit_formatter'
+  "-f RspecJunitFormatter -o results.xml"
+end
+
+namespace :spec do
+  desc "Run all unit and integration tests"
+  task :ci do
+    rspec_out_file = nil
+    sh "bundle exec rspec #{rspec_out_file} spec"
+  end
+
+  desc "Run perf tests"
+  task :perf do
+    rspec_out_file = nil
+    sh "bundle exec ruby ./spec/benchmark/perf_test.rb"
+  end
+
+  desc "Run all tests and generate coverage xml"
+  task :cov do
+    sh "bundle exec rspec #{rspec_out_file} spec"
+  end
+end

data/lib/worker_roulette/foreman.rb

@@ -0,0 +1,71 @@
+module WorkerRoulette
+  class Foreman
+    attr_reader :sender, :namespace, :channel
+
+    LUA_ENQUEUE_WORK_ORDERS = <<-HERE
+      local counter_key       = KEYS[1]
+      local job_board_key     = KEYS[2]
+      local sender_key        = KEYS[3]
+      local channel           = KEYS[4]
+
+      local work_order        = ARGV[1]
+      local job_notification  = ARGV[2]
+      local redis_call        = redis.call
+      local zscore            = 'ZSCORE'
+      local incr              = 'INCR'
+      local zadd              = 'ZADD'
+      local rpush             = 'RPUSH'
+      local publish           = 'PUBLISH'
+      local zcard             = 'ZCARD'
+      local del               = 'DEL'
+
+      local function enqueue_work_orders(work_order, job_notification)
+        redis_call(rpush, sender_key, work_order)
+
+        -- called when a work from a new sender is added
+        if (redis_call(zscore, job_board_key, sender_key) == false) then
+          local count     = redis_call(incr, counter_key)
+          redis_call(zadd, job_board_key, count, sender_key)
+        end
+      end
+
+      enqueue_work_orders(work_order, job_notification)
+    HERE
+
+    def initialize(redis_pool, sender, namespace = nil)
+      @sender     = sender
+      @namespace  = namespace
+      @redis_pool = redis_pool
+      @channel    = namespace || WorkerRoulette::JOB_NOTIFICATIONS
+      @lua        = Lua.new(@redis_pool)
+    end
+
+    def enqueue_work_order(work_order, headers = {}, &callback)
+      work_order = {'headers' => default_headers.merge(headers), 'payload' => work_order}
+      enqueue_work_order_without_headers(work_order, &callback)
+    end
+
+    def enqueue_work_order_without_headers(work_order, &callback)
+      @lua.call(LUA_ENQUEUE_WORK_ORDERS, [counter_key, job_board_key, sender_key, @channel],
+                [WorkerRoulette.dump(work_order),  WorkerRoulette::JOB_NOTIFICATIONS], &callback)
+    end
+
+    def job_board_key
+      @job_board_key ||= WorkerRoulette.job_board_key(@namespace)
+    end
+
+    def counter_key
+      @counter_key ||= WorkerRoulette.counter_key(@namespace)
+    end
+
+    def sender_key
+      @sender_key ||= WorkerRoulette.sender_key(@sender, @namespace)
+    end
+
+    private
+
+    def default_headers
+      Hash['sender' => sender]
+    end
+  end
+end

data/lib/worker_roulette/lua.rb

@@ -0,0 +1,50 @@
+module WorkerRoulette
+  class Lua
+    Thread.main[:worker_roulette_lua_script_cache] = Hash.new
+
+    def initialize(connection_pool)
+      @connection_pool = connection_pool
+    end
+
+    def call(lua_script, keys_accessed = [], args = [], &callback)
+      @connection_pool.with do |redis|
+        evalsha(redis, lua_script, keys_accessed, args, &callback)
+      end
+    end
+
+    def sha(lua_script)
+      Thread.main[:worker_roulette_lua_script_cache][lua_script] ||= Digest::SHA1.hexdigest(lua_script)
+    end
+
+    def cache
+      Thread.main[:worker_roulette_lua_script_cache].dup
+    end
+
+    def clear_cache!
+      Thread.main[:worker_roulette_lua_script_cache] = {}
+    end
+
+    def eval(redis, lua_script, keys_accessed, args, &callback)
+      results = redis.eval(lua_script, keys_accessed.length, *keys_accessed, *args)
+      results.callback(&callback) if callback
+      results.errback  {|err_msg| raise EM::Hiredis::RedisError.new(err_msg)}
+    end
+
+    def evalsha(redis, lua_script, keys_accessed, args, &callback)
+      if redis.class == EM::Hiredis::Client
+        results = redis.evalsha(sha(lua_script), keys_accessed.length, *keys_accessed, *args)
+        results.callback(&callback) if callback
+        results.errback {eval(redis, lua_script, keys_accessed, args, &callback)}
+      else
+        begin
+          results = redis.evalsha(sha(lua_script), keys_accessed, args)
+        rescue Redis::CommandError
+          results = redis.eval(lua_script, keys_accessed, args)
+        ensure
+          return callback.call results if callback
+        end
+      end
+      results
+    end
+  end
+end

data/lib/worker_roulette/tradesman.rb

@@ -0,0 +1,125 @@
+module WorkerRoulette
+  class Tradesman
+    attr_reader :last_sender, :remaining_jobs, :timer
+
+    LUA_DRAIN_WORK_ORDERS = <<-HERE
+      local empty_string      = ""
+      local job_board_key     = KEYS[1]
+      local last_sender_key   = KEYS[2] or empty_string
+      local sender_key        = ARGV[1] or empty_string
+      local redis_call        = redis.call
+      local lock_key_prefix   = "L*:"
+      local lock_value        = 1
+      local ex                = "EX"
+      local nx                = "NX"
+      local get               = "GET"
+      local set               = "SET"
+      local del               = "DEL"
+      local lrange            = "LRANGE"
+      local zrank             = "ZRANK"
+      local zrange            = "ZRANGE"
+      local zrem              = "ZREM"
+      local zcard             = 'ZCARD'
+
+      local function drain_work_orders(job_board_key, last_sender_key, sender_key)
+
+        --kill lock for last_sender_key
+        if last_sender_key ~= empty_string then
+          local last_sender_lock_key = lock_key_prefix .. last_sender_key
+          redis_call(del, last_sender_lock_key)
+        end
+
+        if sender_key == empty_string then
+          sender_key = redis_call(zrange, job_board_key, 0, 0)[1] or empty_string
+
+          -- return if job_board is empty
+          if sender_key == empty_string then
+            return {empty_string, {}, 0}
+          end
+        end
+
+        local lock_key       = lock_key_prefix .. sender_key
+        local was_not_locked = redis_call(set, lock_key, lock_value, ex, 3, nx)
+
+        if was_not_locked then
+          local work_orders    = redis_call(lrange, sender_key, 0, -1)
+          redis_call(del, sender_key)
+
+          redis_call(zrem, job_board_key, sender_key)
+          local remaining_jobs = redis_call(zcard, job_board_key) or 0
+
+          return {sender_key, work_orders, remaining_jobs}
+        else
+          local sender_index    = redis_call(zrank, job_board_key, sender_key)
+          local next_index      = sender_index + 1
+          local next_sender_key = redis_call(zrange, job_board_key, next_index, next_index)[1]
+          if next_sender_key then
+            return drain_work_orders(job_board_key, empty_string, next_sender_key)
+          else
+            -- return if job_board is empty
+            return {empty_string, {}, 0}
+          end
+        end
+      end
+
+      return drain_work_orders(job_board_key, last_sender_key, empty_string)
+    HERE
+
+    def initialize(redis_pool, evented, namespace = nil, polling_time = WorkerRoulette::DEFAULT_POLLING_TIME)
+      @evented        = evented
+      @polling_time   = polling_time
+      @redis_pool     = redis_pool
+      @namespace      = namespace
+      @channel        = namespace || WorkerRoulette::JOB_NOTIFICATIONS
+      @lua            = Lua.new(@redis_pool)
+      @remaining_jobs = 0
+    end
+
+    def wait_for_work_orders(&on_message_callback)
+      return unless on_message_callback
+      work_orders! do |work|
+        on_message_callback.call(work) if work.any?
+        if @evented
+          evented_drain_work_queue!(&on_message_callback)
+        else
+          non_evented_drain_work_queue!(&on_message_callback)
+        end
+      end
+    end
+
+    def work_orders!(&callback)
+      @lua.call(LUA_DRAIN_WORK_ORDERS, [job_board_key, @last_sender], [nil]) do |results|
+        sender_key      = results[0]
+        work_orders     = results[1]
+        @remaining_jobs = results[2]
+        @last_sender    = sender_key.split(':').last
+        work            = work_orders.map {|work_order| WorkerRoulette.load(work_order)}
+        callback.call work if callback
+        work
+      end
+    end
+
+    def job_board_key
+      @job_board_key ||= WorkerRoulette.job_board_key(@namespace)
+    end
+
+    private
+
+    def evented_drain_work_queue!(&on_message_callback)
+      if remaining_jobs > 0
+        EM.next_tick {wait_for_work_orders(&on_message_callback)}
+      else
+        EM.add_timer(@polling_time) { wait_for_work_orders(&on_message_callback) }
+      end
+    end
+
+    def non_evented_drain_work_queue!(&on_message_callback)
+      if remaining_jobs > 0
+        wait_for_work_orders(&on_message_callback)
+      else
+        sleep 2
+        wait_for_work_orders(&on_message_callback)
+      end
+    end
+  end
+end

data/lib/worker_roulette/version.rb

@@ -0,0 +1,3 @@
+module WorkerRoulette
+  VERSION = '0.1.11'
+end

data/lib/worker_roulette.rb

@@ -0,0 +1,96 @@
+require "worker_roulette/version"
+require 'oj'
+require 'redis'
+require 'hiredis'
+require 'em-hiredis'
+require 'connection_pool'
+require "digest/sha1"
+
+Dir[File.join(File.dirname(__FILE__),'worker_roulette','**','*.rb')].sort.each { |file| require file.gsub(".rb", "")}
+
+module WorkerRoulette
+  class WorkerRoulette
+    JOB_BOARD = "job_board"
+    JOB_NOTIFICATIONS = "new_job_ready"
+    DEFAULT_POLLING_TIME = 2
+
+    def self.dump(obj)
+      Oj.dump(obj)
+    rescue Oj::ParseError => e
+      {'error' => e, 'unparsable_string' => obj}
+    end
+
+    def self.load(json)
+      Oj.load(json)
+    rescue Oj::ParseError => e
+      {'error' => e, 'unparsable_string' => obj}
+    end
+
+    def self.job_board_key(namespace = nil)
+      "#{namespace + ':' if namespace}#{WorkerRoulette::JOB_BOARD}"
+    end
+
+    def self.sender_key(sender, namespace = nil)
+      "#{namespace + ':' if namespace}#{sender}"
+    end
+
+    def self.counter_key(sender, namespace = nil)
+      "#{namespace + ':' if namespace}counter_key"
+    end
+
+    def self.start(config = {})
+      instance = new(config)
+      instance
+    end
+
+    private_class_method :new
+
+    def initialize(config = {})
+      @redis_config               = { host: 'localhost', port: 6379, db: 14, driver: :hiredis, timeout: 5, evented: false, pool_size: 10 , polling_time: DEFAULT_POLLING_TIME}.merge(config)
+      @pool_config                = Hash[size: @redis_config.delete(:pool_size), timeout: @redis_config.delete(:timeout)]
+      @evented                    = @redis_config.delete(:evented)
+      @polling_time               = @redis_config.delete(:polling_time)
+
+      @foreman_connection_pool    = ConnectionPool.new(@pool_config) {new_redis}
+      @tradesman_connection_pool  = ConnectionPool.new(@pool_config) {new_redis}
+    end
+
+    def foreman(sender, namespace = nil)
+      raise "WorkerRoulette not Started" unless @foreman_connection_pool
+      Foreman.new(@foreman_connection_pool, sender, namespace)
+    end
+
+    def tradesman(namespace = nil, polling_time = DEFAULT_POLLING_TIME)
+      raise "WorkerRoulette not Started" unless @tradesman_connection_pool
+      Tradesman.new(@tradesman_connection_pool, @evented, namespace, polling_time || @polling_time)
+    end
+
+    def tradesman_connection_pool
+      @tradesman_connection_pool
+    end
+
+    def pool_size
+      (@pool_config ||= {})[:size]
+    end
+
+    def redis_config
+      (@redis_config ||= {}).dup
+    end
+
+    def polling_time
+      @polling_time
+    end
+
+    private
+
+    def new_redis
+      if @evented
+        require 'eventmachine'
+        redis = EM::Hiredis::Client.new(@redis_config[:host], @redis_config[:port], @redis_config[:password], @redis_config[:db])
+        redis.connect
+      else
+        Redis.new(@redis_config)
+      end
+    end
+  end
+end

data/spec/benchmark/irb_demo_runner.rb

@@ -0,0 +1,39 @@
+require 'worker_roulette'
+
+def pub(iterations)
+  instance = WorkerRoulette::WorkerRoulette.start(evented: false)
+  work_order = {'ding dong' => "hello_foreman_" * 100}
+  iterations.times do |iteration|
+    sender = 'sender_' + (30_000 * rand).to_i.to_s
+    foreman = instance.foreman(sender, 'good_channel')
+    foreman.enqueue_work_order(work_order)
+    puts "published: #{iteration}" if iteration % 10_000 == 0
+  end
+end
+
+def sub(iterations)
+  instance = WorkerRoulette::WorkerRoulette.start(evented: true)
+  @tradesman = instance.tradesman('good_channel')
+  @received = 0
+  @tradesman.wait_for_work_orders do |work|
+    @received += work.length
+    puts @received if @received % (iterations / 10) == 0
+  end
+end
+
+def start(action, iterations = 1_000_000)
+  EM.kqueue = true
+  socket_max = 50_000
+  EventMachine.set_descriptor_table_size(socket_max)
+
+  EM.run do
+    Signal.trap("INT") {
+      EM.stop_event_loop
+    }
+    Signal.trap("TERM") {
+      EM.stop_event_loop
+    }
+
+    self.send(action, iterations)
+  end
+end