backburner 1.3.0 → 1.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ba786ac1dbb767afc923b6da0ef80e525043ad20
-  data.tar.gz: 200863f6e8c0be7731e1448c9c3285db82f3b31f
+  metadata.gz: ffcc49440304bb0d25a3710a3d94442fd1881f51
+  data.tar.gz: 2e9318efc9a04979135fecc8edfce4e8c8912386
 SHA512:
-  metadata.gz: 869c7ec413947f7b58c215aae770aea0826a005b414804d39f356c264df912c41d5ee02f5516d7bd357b937b5d7f48a5d2ac92faa0d450f46ae735bf54217f1b
-  data.tar.gz: dfcac3547f1a52ea5f6b955a996fd05fa6375b0bef0e9a2b196c363dd913d09885ff998dd4ae54e7305ff8835c9ebd477eac88a2d6ee8a8b0e1fc96953ee87b2
+  metadata.gz: caa6aa25343e41b2e8097be38d6b5679f304b9f417f2b0559e085caa2165660fc3ab132f5a920a49360faf16629b84b2d1b740ddf8332b0106df0437458a6f40
+  data.tar.gz: 457cb6f9ce73c900e0c54face79f78c8f0f6a89b1462580208f5afa9450f17c717fd7b2dd5701299addb334e260e55baa5c5360583867268c9f4ea921c94e38b

data/.travis.yml

@@ -19,3 +19,4 @@ gemfile: Gemfile
 notifications:
   recipients:
     - nesquena@gmail.com
+    - therealdave.myron@gmail.com

data/CHANGELOG.md

@@ -1,6 +1,10 @@
 # CHANGELOG
 
-## Version 1.3.0 (February 05 2015)
+## Version 1.3.1 (April 21 2016)
+
+* Addition of thread-pool-based concurrency (@contentfree)
+
+## Version 1.3.0 (February 05 2016)
 
 * Enqueue command now responds with beanstalk response details
 

data/HOOKS.md

@@ -5,7 +5,7 @@ In many cases you can use a hook rather than mess around with Backburner's inter
 
 ## Job Hooks
 
-Hooks are transparently adapted from [Resque](https://github.com/defunkt/resque/blob/master/docs/HOOKS.md), so
+Hooks are transparently adapted from [Resque](https://github.com/resque/resque/blob/master/docs/HOOKS.md), so
 if you are familiar with their hook API, now you can use nearly the same ones with beanstalkd and backburner!
 
 There are a variety of hooks available that are triggered during the lifecycle of a job:
@@ -56,7 +56,31 @@ class SomeJob
 end
 ```
 
-You can also setup modules to create compose-able and reusable hooks for your jobs.
+You can also setup modules to create compose-able and reusable hooks for your jobs. For example:
+
+```ruby
+module LoggedJob
+  def before_perform_log_job(*args)
+    Logger.info "About to perform #{self} with #{args.inspect}"
+  end
+end
+
+module BuriedJob
+  def on_failure_bury(e, *args)
+    Logger.info "Performing #{self} caused an exception (#{e}). Retrying..."
+    self.bury
+  end
+end
+
+class MyJob
+  extend LoggedJob
+  extend BuriedJob
+
+  def self.perform(*args)
+    # ...
+  end
+end
+```
 
 ## Worker Hooks
 

data/README.md

@@ -126,8 +126,8 @@ The key options available are:
 
 ## Breaking Changes
 
-Since **v0.4.0**: Jobs used to be placed into default queues based on the name of the class enqueuing i.e NewsletterJob would
-be put into a 'newsletter-job' queue. After 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs"
+Before **v0.4.0**: Jobs were placed into default queues based on the name of the class creating the queue. i.e NewsletterJob would
+be put into a 'newsletter-job' queue. As of 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs"
 unless otherwise specified.
 
 ## Usage
@@ -402,6 +402,7 @@ By default, Backburner comes with the following workers built-in:
 | `Backburner::Workers::Simple` | Single threaded, no forking worker. Simplest option. |
 | `Backburner::Workers::Forking` | Basic forking worker that manages crashes and memory bloat. |
 | `Backburner::Workers::ThreadsOnFork` | Forking worker that utilizes threads for concurrent processing. |
+| `Backburner::Workers::Threading` | Utilizes thread pools for concurrent processing. |
 
 You can select the default worker for processing with:
 
@@ -423,12 +424,23 @@ or through associated rake tasks with:
 $ QUEUE=newsletter-sender,push-message THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work
 ```
 
+When running on MRI or another Ruby implementation with a Global Interpreter Lock (GIL), do not be surprised if you're unable to saturate multiple cores, even with the threads_on_fork worker. To utilize multiple cores, you must run multiple worker processes.
+
+Additional concurrency strategies will hopefully be contributed in the future.
+If you are interested in helping out, please let us know.
+
+#### More info: Threads on Fork Worker
+
 For more information on the threads_on_fork worker, check out the
 [ThreadsOnFork Worker](https://github.com/nesquena/backburner/wiki/ThreadsOnFork-worker) documentation. Please note that the `ThreadsOnFork` worker does not work on Windows due to its lack of `fork`.
 
+#### More info: Threading Worker (thread-pool-based)
 
-Additional workers such as individual `threaded` and `forking` strategies will hopefully be contributed in the future.
-If you are interested in helping out, please let us know.
+Configuration options for the Threading worker are similar to the threads_on_fork worker, sans the garbage option. When running via the `backburner` CLI, it's simplest to provide the queue names and maximum number of threads in the format "{queue name}:{max threads in pool}[,{name}:{threads}]":
+
+```
+$ bundle exec backburner -q queue1:4,queue2:4  # and then other options, like environment, pidfile, app root, etc. See docs for the CLI
+```
 
 ### Default Queues
 

data/backburner.gemspec

@@ -18,8 +18,10 @@ Gem::Specification.new do |s|
 
   s.add_runtime_dependency 'beaneater', '~> 1.0'
   s.add_runtime_dependency 'dante', '> 0.1.5'
+  s.add_runtime_dependency 'concurrent-ruby', '~> 1.0.1'
 
   s.add_development_dependency 'rake'
   s.add_development_dependency 'minitest', '3.2.0'
   s.add_development_dependency 'mocha'
+  s.add_development_dependency 'byebug'
 end

data/lib/backburner.rb

@@ -13,6 +13,7 @@ require 'backburner/worker'
 require 'backburner/workers/simple'
 require 'backburner/workers/forking'
 require 'backburner/workers/threads_on_fork'
+require 'backburner/workers/threading'
 require 'backburner/queue'
 
 module Backburner

data/lib/backburner/queue.rb

@@ -54,7 +54,7 @@ module Backburner
         end
       end
 
-      # Returns or assigns queue parallel active jobs limit (only ThreadsOnFork Worker)
+      # Returns or assigns queue parallel active jobs limit (only ThreadsOnFork and Threading workers)
       #
       # @example
       #   queue_jobs_limit 5
@@ -82,7 +82,7 @@ module Backburner
         end
       end
 
-      # Returns or assigns queue retry limit (only ThreadsOnFork Worker)
+      # Returns or assigns queue retry limit (only ThreadsOnFork worker)
       #
       # @example
       #   queue_retry_limit 6

data/lib/backburner/tasks.rb

@@ -5,16 +5,14 @@ namespace :backburner do
   # QUEUE=foo,bar,baz rake backburner:work
   desc "Start backburner worker using default worker"
   task :work => :environment do
-    queues = (ENV["QUEUE"] ? ENV["QUEUE"].split(',') : nil) rescue nil
-    Backburner.work queues
+    Backburner.work get_queues
   end
 
   namespace :simple do
     # QUEUE=foo,bar,baz rake backburner:simple:work
     desc "Starts backburner worker using simple processing"
     task :work => :environment do
-      queues = (ENV["QUEUE"] ? ENV["QUEUE"].split(',') : nil) rescue nil
-      Backburner.work queues, :worker => Backburner::Workers::Simple
+      Backburner.work get_queues, :worker => Backburner::Workers::Simple
     end
   end # simple
 
@@ -22,8 +20,7 @@ namespace :backburner do
     # QUEUE=foo,bar,baz rake backburner:forking:work
     desc "Starts backburner worker using fork processing"
     task :work => :environment do
-      queues = (ENV["QUEUE"] ? ENV["QUEUE"].split(',') : nil) rescue nil
-      Backburner.work queues, :worker => Backburner::Workers::Forking
+      Backburner.work get_queues, :worker => Backburner::Workers::Forking
     end
   end # forking
 
@@ -32,12 +29,26 @@ namespace :backburner do
     # twitter tube will have 10 threads, garbage after 5k executions and retry 5 times.
     desc "Starts backburner worker using threads_on_fork processing"
     task :work => :environment do
-      queues = (ENV["QUEUE"] ? ENV["QUEUE"].split(',') : nil) rescue nil
       threads = ENV['THREADS'].to_i
       garbage = ENV['GARBAGE'].to_i
       Backburner::Workers::ThreadsOnFork.threads_number = threads if threads > 0
       Backburner::Workers::ThreadsOnFork.garbage_after  = garbage if garbage > 0
-      Backburner.work queues, :worker => Backburner::Workers::ThreadsOnFork
+      Backburner.work get_queues, :worker => Backburner::Workers::ThreadsOnFork
     end
   end # threads_on_fork
-end
+
+  namespace :threading do
+    # QUEUE=twitter:10,parse_page,send_mail,verify_bithday THREADS=2 rake backburner:threading:work
+    # twitter tube will have 10 threads
+    desc "Starts backburner worker using threading processing"
+    task :work => :environment do
+      threads = ENV['THREADS'].to_i
+      Backburner::Workers::Threading.threads_number = threads if threads > 0
+      Backburner.work get_queues, :worker => Backburner::Workers::Threading
+    end
+  end # threads_on_fork
+
+  def get_queues
+    (ENV["QUEUE"] ? ENV["QUEUE"].split(',') : nil) rescue nil
+  end
+end

data/lib/backburner/version.rb

@@ -1,3 +1,3 @@
 module Backburner
-  VERSION = "1.3.0"
+  VERSION = "1.3.1"
 end

data/lib/backburner/workers/threading.rb

@@ -0,0 +1,129 @@
+require 'concurrent'
+
+module Backburner
+  module Workers
+    class Threading < Worker
+      class << self
+        attr_accessor :shutdown
+        attr_accessor :threads_number
+      end
+
+      # Custom initializer just to set @tubes_data
+      def initialize(*args)
+        @tubes_data = {}
+        super
+        self.process_tube_options
+      end
+
+      # Used to prepare job queues before processing jobs.
+      # Setup beanstalk tube_names and watch all specified tubes for jobs.
+      #
+      # @raise [Beaneater::NotConnected] If beanstalk fails to connect.
+      # @example
+      #   @worker.prepare
+      #
+      def prepare
+        self.tube_names.map! { |name| expand_tube_name(name)  }.uniq!
+        log_info "Working #{tube_names.size} queues: [ #{tube_names.join(', ')} ]"
+        @thread_pools = {}
+        @tubes_data.each do |name, config|
+          max_threads = (config[:threads] || self.class.threads_number || ::Concurrent.processor_count).to_i
+          @thread_pools[name] = (::Concurrent::ThreadPoolExecutor.new(min_threads: 1, max_threads: max_threads))
+        end
+      end
+
+      # Starts processing new jobs indefinitely.
+      # Primary way to consume and process jobs in specified tubes.
+      #
+      # @example
+      #   @worker.start
+      #
+      def start(wait=true)
+        prepare
+
+        @thread_pools.each do |tube_name, pool|
+          pool.max_length.times do
+            # Create a new connection and set it up to listen on this tube name
+            connection = new_connection.tap{ |conn| conn.tubes.watch!(tube_name) }
+            connection.on_reconnect = lambda { |conn| conn.tubes.watch!(tube_name) }
+
+            # Make it work jobs using its own connection per thread
+            pool.post(connection) { |connection|
+              loop {
+                begin
+                  work_one_job(connection)
+
+                rescue => e
+                  log_error("Exception caught in thread pool loop. Continuing. -> #{e.message}\nBacktrace: #{e.backtrace}")
+                end
+              }
+            }
+          end
+        end
+
+        wait_for_shutdown! if wait
+      end
+
+      # FIXME: We can't use this on_reconnect method since we don't know which thread
+      # pool the connection belongs to (and therefore we can't re-watch the right tubes).
+      # However, we set the individual connections' on_reconnect method in #start
+      # def on_reconnect(conn)
+      #   watch_tube(@watching_tube, conn) if @watching_tube
+      # end
+
+      # Process the special tube_names of Threading worker:
+      #   The format is tube_name:custom_threads_limit
+      #
+      # @example
+      #    process_tube_names(['foo:10', 'lol'])
+      #    => ['foo', lol']
+      def process_tube_names(tube_names)
+        names = compact_tube_names(tube_names)
+        if names.nil?
+          nil
+        else
+          names.map do |name|
+            data = name.split(":")
+            tube_name = data.first
+            threads_number = data[1].empty? ? nil : data[1].to_i rescue nil
+            @tubes_data[expand_tube_name(tube_name)] = {
+              :threads => threads_number
+            }
+            tube_name
+          end
+        end
+      end
+
+      # Process the tube settings
+      # This overrides @tubes_data set by process_tube_names method. So a tube has name 'super_job:5'
+      # and the tube class has setting queue_jobs_limit 10, the result limit will be 10
+      # If the tube is known by existing beanstalkd queue, but not by class - skip it
+      #
+      def process_tube_options
+        Backburner::Worker.known_queue_classes.each do |queue|
+          next if @tubes_data[expand_tube_name(queue)].nil?
+          queue_settings = {
+            :threads => queue.queue_jobs_limit
+          }
+          @tubes_data[expand_tube_name(queue)].merge!(queue_settings){|k, v1, v2| v2.nil? ? v1 : v2 }
+        end
+      end
+
+      # Wait for the shutdown signel
+      def wait_for_shutdown!
+        while !self.class.shutdown do
+          sleep 0.5
+        end
+
+        # Shutting down
+        # FIXME: Shut down each thread's connection
+        @thread_pools.each { |name, pool| pool.kill }
+      end
+
+      def shutdown
+        Backburner::Workers::Threading.shutdown = true
+        super
+      end
+    end # Threading
+  end # Workers
+end # Backburner

data/lib/backburner/workers/threads_on_fork.rb

@@ -69,7 +69,6 @@ module Backburner
       # Custom initializer just to set @tubes_data
       def initialize(*args)
         @tubes_data = {}
-        @mutex = Mutex.new
         super
         self.process_tube_options
       end

data/test/workers/simple_worker_test.rb

@@ -2,7 +2,7 @@ require File.expand_path('../../test_helper', __FILE__)
 require File.expand_path('../../fixtures/test_jobs', __FILE__)
 require File.expand_path('../../fixtures/hooked', __FILE__)
 
-describe "Backburner::Workers::Basic module" do
+describe "Backburner::Workers::Simple module" do
   before do
     Backburner.default_queues.clear
     @worker_class = Backburner::Workers::Simple

data/test/workers/threading_worker_test.rb

@@ -0,0 +1,68 @@
+require File.expand_path('../../test_helper', __FILE__)
+require File.expand_path('../../fixtures/test_jobs', __FILE__)
+require File.expand_path('../../fixtures/hooked', __FILE__)
+
+describe "Backburner::Workers::Threading worker" do
+  before do
+    Backburner.default_queues.clear
+    @worker_class = Backburner::Workers::Threading
+  end
+
+  describe "for prepare method" do
+    it "should make tube names array always unique to avoid duplication" do
+      worker = @worker_class.new(["foo", "demo.test.foo"])
+      worker.prepare
+      assert_equal ["demo.test.foo"], worker.tube_names
+    end
+
+    it 'creates a thread pool per queue' do
+      worker = @worker_class.new(%w(foo bar))
+      out = capture_stdout { worker.prepare }
+      assert_equal 2, worker.instance_variable_get("@thread_pools").keys.size
+    end
+
+    it 'uses Concurrent.processor_count if no custom thread count is provided' do
+      worker = @worker_class.new("foo")
+      out = capture_stdout { worker.prepare }
+      assert_equal ::Concurrent.processor_count, worker.instance_variable_get("@thread_pools")["demo.test.foo"].max_length
+    end
+  end # prepare
+
+  describe "for process_tube_names method" do
+    it "should interpret the job_name:threads_limit format" do
+      worker = @worker_class.new(["foo:4"])
+      assert_equal ["foo"], worker.tube_names
+    end
+
+    it "should interpret correctly even if missing values" do
+      tubes = %W(foo1:2 foo2)
+      worker = @worker_class.new(tubes)
+      assert_equal %W(foo1 foo2), worker.tube_names
+    end
+
+    it "should store interpreted values correctly" do
+      tubes = %W(foo1 foo2:2)
+      worker = @worker_class.new(tubes)
+      assert_equal({
+        "demo.test.foo1" => { :threads => nil },
+        "demo.test.foo2" => { :threads => 2 }
+      }, worker.instance_variable_get("@tubes_data"))
+    end
+  end # process_tube_names
+
+  describe 'working a queue' do
+    before do
+      @worker = @worker_class.new(["foo:3"])
+      capture_stdout { @worker.prepare }
+    end
+
+    it 'runs work_on_job per thread' do
+      clear_jobs!("foo")
+      job_count=10
+      job_count.times { @worker_class.enqueue TestJob, [1, 0], :queue => "foo" } # TestJob adds the given arguments together and then to $worker_test_count
+      @worker.start(false) # don't wait for shutdown
+      sleep 0.5 # Wait for threads to do their work
+      assert_equal job_count, $worker_test_count
+    end
+  end # working a queue
+end

metadata

@@ -1,55 +1,69 @@
 --- !ruby/object:Gem::Specification
 name: backburner
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.3.1
 platform: ruby
 authors:
 - Nathan Esquenazi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-02-05 00:00:00.000000000 Z
+date: 2016-10-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: beaneater
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.0'
 - !ruby/object:Gem::Dependency
   name: dante
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>'
+    - - ">"
       - !ruby/object:Gem::Version
         version: 0.1.5
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>'
+    - - ">"
       - !ruby/object:Gem::Version
         version: 0.1.5
+- !ruby/object:Gem::Dependency
+  name: concurrent-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.0.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.0.1
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
@@ -70,14 +84,28 @@ dependencies:
   name: mocha
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: byebug
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 description: Beanstalk background job processing made easy
@@ -88,8 +116,8 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
-- .travis.yml
+- ".gitignore"
+- ".travis.yml"
 - CHANGELOG.md
 - CONTRIBUTING.md
 - Gemfile
@@ -124,6 +152,7 @@ files:
 - lib/backburner/worker.rb
 - lib/backburner/workers/forking.rb
 - lib/backburner/workers/simple.rb
+- lib/backburner/workers/threading.rb
 - lib/backburner/workers/threads_on_fork.rb
 - test/async_proxy_test.rb
 - test/back_burner_test.rb
@@ -144,6 +173,7 @@ files:
 - test/worker_test.rb
 - test/workers/forking_worker_test.rb
 - test/workers/simple_worker_test.rb
+- test/workers/threading_worker_test.rb
 - test/workers/threads_on_fork_worker_test.rb
 homepage: http://github.com/nesquena/backburner
 licenses:
@@ -155,12 +185,12 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
@@ -189,4 +219,5 @@ test_files:
 - test/worker_test.rb
 - test/workers/forking_worker_test.rb
 - test/workers/simple_worker_test.rb
+- test/workers/threading_worker_test.rb
 - test/workers/threads_on_fork_worker_test.rb