hirefire 0.1.0 → 0.1.1

data/README.md

@@ -1,7 +1,7 @@
 HireFire - The Heroku Worker Manager
 ====================================
 
-**HireFire automatically "hires" and "fires" (aka "scales") Delayed Job workers on Heroku**. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases.
+**HireFire automatically "hires" and "fires" (aka "scales") [Delayed Job](https://github.com/collectiveidea/delayed_job) and [Resque](https://github.com/defunkt/resque) workers on Heroku**. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases.
 
 **Low traffic example** say we have a small application that doesn't process for more than 2 hours in the background a month. Meanwhile, your worker is basically just idle the rest of the 718 hours in that month. Keeping that idle worker running costs $36/month ($0.05/hour). But, for the resources you're actually **making use of** (2 hours a month), you should be paying $0.10/month, not $36/month. This is what HireFire is for.
 
@@ -25,10 +25,11 @@ A painless process. In a Ruby on Rails environment you would do something like t
 **Rails.root/Gemfile**
 
     gem 'rails'
-    gem 'delayed_job'
+    # gem 'delayed_job' # uncomment this line if you use Delayed Job
+    # gem 'resque'      # uncomment this line if you use Resque
     gem 'hirefire'
 
-**(The order is important: Delayed Job > HireFire)**
+**(The order is important: "Delayed Job" / "Resque" > HireFire)**
 
 Be sure to add the following Heroku environment variables so HireFire can manage your workers.
 
@@ -41,7 +42,8 @@ And that's it. Next time you deploy to [Heroku](http://heroku.com/) it'll automa
 **Rails.root/config/initializers/hirefire.rb**
 
     HireFire.configure do |config|
-      config.max_workers      = 5 # default is 1
+      config.environment      = nil # default in production is :heroku. default in development is :noop
+      config.max_workers      = 5   # default is 1
       config.job_worker_ratio = [
           { :jobs => 1,   :workers => 1 },
           { :jobs => 15,  :workers => 2 },
@@ -61,35 +63,80 @@ Basically what it comes down to is that we say **NEVER** to hire more than 5 wor
 
 Once all the jobs in the queue have been processed, it'll fire (shut down) all the workers and start with a single worker the next time a new job gets queued. And then the next time the queue hits 15 jobs mark, in which case the single worker isn't fast enough on it's own, it'll spin up the 2nd worker again.
 
+*If you prefer a more functional way of defining your job/worker ratio, you could use the following notation style:*
+
+    HireFire.configure do |config|
+      config.max_workers = 5
+      config.job_worker_ratio = [
+        { :when => lambda {|jobs| jobs < 15 }, :workers => 1 },
+        { :when => lambda {|jobs| jobs < 35 }, :workers => 2 },
+        { :when => lambda {|jobs| jobs < 60 }, :workers => 3 },
+        { :when => lambda {|jobs| jobs < 80 }, :workers => 4 }
+      ]
+    end
+
+The above notation is slightly different, since now you basically define how many workers to hire when `jobs < n`. So for example if there are 80 or more jobs, it'll hire the `max_workers` amount, which is `5` in the above example. If you change the `max_workers = 5` to `max_workers = 10`, then if there are 80 or more jobs queued, it'll go from 4 to 10 workers.
+
 
 In a non-Ruby on Rails environment
 ----------------------------------
 
-Almost the same setup, except that you have to initialize HireFire yourself after Delayed Job is done loading.
+Almost the same setup, except that you have to initialize HireFire yourself after Delayed Job or Resque is done loading.
 
     require 'delayed_job'
-    require 'hirefire'
+    # require 'delayed_job' # uncomment this line if you use Delayed Job
+    # require 'resque'      # uncomment this line if you use Resque
     HireFire::Initializer.initialize!
 
-**(Again, the order is important: Delayed Job > HireFire)**
+**(Again, the order is important: "Delayed Job" / "Resque" > HireFire)**
 
 If all goes well you should see a message similar to this when you boot your application:
 
     [HireFire] Delayed::Backend::ActiveRecord::Job detected!
 
 
-Mapper Support
+Worker / Mapper Support
 --------------
 
-* [ActiveRecord ORM](https://github.com/rails/rails/tree/master/activerecord)
-* [Mongoid ODM](https://github.com/mongoid/mongoid) (using [delayed_job_mongoid](https://github.com/collectiveidea/delayed_job_mongoid))
+HireFire currently works with the following worker and mapper libraries:
 
+- [Delayed Job](https://github.com/collectiveidea/delayed_job)
+  - [ActiveRecord ORM](https://github.com/rails/rails/tree/master/activerecord)
+  - [Mongoid ODM](https://github.com/mongoid/mongoid) (using [delayed_job_mongoid](https://github.com/collectiveidea/delayed_job_mongoid))
 
-Worker Support
---------------
+- [Resque](https://github.com/defunkt/resque)
+  - [Redis](https://github.com/ezmobius/redis-rb)
+
+
+Frequently Asked Questions
+--------------------------
+
+- **Question:** *Does it start workers immediately after a job gets queued?*
+  - **Answer:** Yes, once a new job gets queued it'll immediately calculate the amount of workers that are required and hire them accordingly.
+
+- **Question:** *Does it stop workers immediately when there are no jobs to be processed?*
+  - **Answer:** Yes, every worker has been made self-aware to see this. Once there are no jobs to be processed, all workers will immediately be fired (shut down). *For example, if you have no jobs in the queue, and you start cranking up your Workers via Heroku's web ui, once the worker spawns and sees it has nothing to do, it'll immediately shut itself down.*
+
+- **Question:** *How does this save me money?*
+  - **Answer:** According to Heroku's documentation, Workers (same as Dynos), are prorated to the second. *For example, say that 10 jobs get queued and a worker is spawned to process them and takes about 1 minute to do so and then shuts itself down, theoretically you only pay $0.0008.*
+
+- **Question:** *With Delayed Job you can set the :run_at to a time in the future.*
+  - **Answer:** Unfortunately since we cannot spawn a monitoring process on the Heroku platform, HireFire will not hire workers until a job gets queued. This means that if you set the :run_at time a few minutes in the future, and these few minutes pass, the job will not be processed until a new job gets queued which triggers the chain of events. (Best to avoid using `run_at` with Delayed Job when using HireFire unless you have a mid-high traffic web application in which cause HireFire gets triggered enough times)
+
+- **Question:** *If a job is set to run at a time in the future, will workers remain hired to wait for this job to be "processable"?*
+  - **Answer:** No, because if you enqueue a job to run 3 hours from the time it was enqueued, you might have workers doing nothing the coming 3 hours. Best to avoid scheduling jobs to be processed in the future.
+
+- **Question:** *Will it scale down workers from, for example, 5 to 4?*
+  - **Answer:** No, I have consciously chosen not to do that for 2 reasons:
+      1. There is no way to tell which worker is currently processing a job, so it might fire a worker that was busy, causing the job to be exit during the process.
+      2. Does it really matter? Your jobs will be processed faster, and once the queue is completely empty, all workers will be fire anyway. (You could call this a feature! Since 5 jobs process faster than 4, but the cost remains the same cause it's all pro-rated to the second)
+
+- **Question:** *Will running jobs concurrently (with multiple Worker) cost more?*
+  - **Answer:** Actually, no. Since worker's are pro-rated to the second, the moment you hire 3 workers, it costs 3 times more, but it also processes 3 times faster. You could also let 1 worker process all the jobs rather than 3, but that means it'll still cost the same amount as when you hire 3 workers, since it takes 3 times longer to process.
+
+- **Question:** *Can I process jobs faster with HireFire?*
+  - **Answer:** When you run multiple jobs concurrently, you can speed up your processing dramatically. *Normally you wouldn't set the workers to 10 for example, but with HireFire you can tell it to Hire 10 workers when there are 50 jobs (would normally be overkill and cost you A LOT of money) but since (see Q/A above) Workers are pro-rated to the second, and HireFire immediately fires all workers once all the jobs in the queue have been processed, it makes no different whether you have a single worker processing 50 jobs, or 5 workers, or even 10 workers. It processes 10 times faster, but costs the same.*
 
-Currently only [Delayed Job](https://github.com/collectiveidea/delayed_job) with either [ActiveRecord ORM](https://github.com/rails/rails/tree/master/activerecord) and [Mongoid ODM](https://github.com/mongoid/mongoid).
-Might have plans to implement this for other workers in the future.
 
 
 Other potentially interesting gems

data/hirefire.gemspec

@@ -12,17 +12,8 @@ Gem::Specification.new do |gem|
   gem.authors     = 'Michael van Rooijen'
   gem.email       = 'meskyanichi@gmail.com'
   gem.homepage    = 'http://rubygems.org/gems/hirefire'
-  gem.summary     = 'HireFire automatically "hires" and "fires" (aka "scales") Delayed Job workers on Heroku.'
-  gem.description = <<-EOS
-                      HireFire automatically "hires" and "fires" (aka "scales") Delayed Job workers on Heroku.
-                      When there are no queue jobs, HireFire will fire (shut down) all workers. If there are
-                      queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired
-                      depends on the amount of queued jobs (the ratio can be configured by you). HireFire
-                      is great for both high, mid and low traffic applications. It can save you a lot of
-                      money by only hiring workers when there are pending jobs, and then firing them again
-                      once all the jobs have been processed. It's also capable to dramatically reducing
-                      processing time by automatically hiring more workers when the queue size increases.
-                    EOS
+  gem.summary     = %|HireFire automatically "hires" and "fires" (aka "scales") Delayed Job and Resque workers on Heroku.|
+  gem.description = %|HireFire automatically "hires" and "fires" (aka "scales") Delayed Job and Resque workers on Heroku. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases.|
 
   ##
   # Files and folder that need to be compiled in to the Ruby Gem

data/lib/hirefire.rb

@@ -5,18 +5,19 @@ module HireFire
   ##
   # HireFire constants
   LIB_PATH         = File.dirname(__FILE__)
-  FREELANCER_PATH  = File.join(LIB_PATH,        'hirefire')
-  ENVIRONMENT_PATH = File.join(FREELANCER_PATH, 'environment')
-  BACKEND_PATH     = File.join(FREELANCER_PATH, 'backend')
+  HIREFIRE_PATH    = File.join(LIB_PATH,      'hirefire')
+  ENVIRONMENT_PATH = File.join(HIREFIRE_PATH, 'environment')
+  BACKEND_PATH     = File.join(HIREFIRE_PATH, 'backend')
+  WORKERS_PATH     = File.join(HIREFIRE_PATH, 'workers')
 
   ##
   # HireFire namespace
-  autoload :Configuration, File.join(FREELANCER_PATH, 'configuration')
-  autoload :Environment,   File.join(FREELANCER_PATH, 'environment')
-  autoload :Initializer,   File.join(FREELANCER_PATH, 'initializer')
-  autoload :Backend,       File.join(FREELANCER_PATH, 'backend')
-  autoload :Logger,        File.join(FREELANCER_PATH, 'logger')
-  autoload :Version,       File.join(FREELANCER_PATH, 'version')
+  autoload :Configuration, File.join(HIREFIRE_PATH, 'configuration')
+  autoload :Environment,   File.join(HIREFIRE_PATH, 'environment')
+  autoload :Initializer,   File.join(HIREFIRE_PATH, 'initializer')
+  autoload :Backend,       File.join(HIREFIRE_PATH, 'backend')
+  autoload :Logger,        File.join(HIREFIRE_PATH, 'logger')
+  autoload :Version,       File.join(HIREFIRE_PATH, 'version')
 
   ##
   # HireFire::Environment namespace
@@ -27,11 +28,31 @@ module HireFire
     autoload :Noop,   File.join(ENVIRONMENT_PATH, 'noop')
   end
 
+  ##
+  # HireFire::Workers namespace
+  module Workers
+    autoload :DelayedJob, File.join(WORKERS_PATH, 'delayed_job')
+    autoload :Resque,     File.join(WORKERS_PATH, 'resque')
+  end
+
   ##
   # HireFire::Backend namespace
   module Backend
-    autoload :ActiveRecord, File.join(BACKEND_PATH, 'active_record')
-    autoload :Mongoid,      File.join(BACKEND_PATH, 'mongoid')
+    DELAYED_JOB_PATH = File.join(BACKEND_PATH, 'delayed_job')
+    RESQUE_PATH      = File.join(BACKEND_PATH, 'resque')
+
+    ##
+    # HireFire::Backend::DelayedJob namespace
+    module DelayedJob
+      autoload :ActiveRecord, File.join(DELAYED_JOB_PATH, 'active_record')
+      autoload :Mongoid,      File.join(DELAYED_JOB_PATH, 'mongoid')
+    end
+
+    ##
+    # HireFire::Backend::Resque namespace
+    module Resque
+      autoload :Redis, File.join(RESQUE_PATH, 'redis')
+    end
   end
 
   ##
@@ -79,7 +100,12 @@ end
 # so that the developer doesn't have to manually invoke it from an initializer file
 #
 # Users not using Ruby on Rails will have to run "HireFire::Initializer.initialize!"
-# in their application manually, after loading Delayed Job and the desired mapper (ActiveRecord or Mongoid)
+# in their application manually, after loading the worker library (either "Delayed Job" or "Resque")
+# and the desired mapper (ActiveRecord, Mongoid or Redis)
 if defined?(Rails)
-  require File.join(HireFire::FREELANCER_PATH, 'railtie')
+  if Rails.version >= '3.0.0'
+    require File.join(HireFire::HIREFIRE_PATH, 'railtie')
+  else
+    HireFire::Initializer.initialize!
+  end
 end

data/lib/hirefire/backend.rb

@@ -4,17 +4,32 @@ module HireFire
   module Backend
 
     ##
-    # Load the correct module (ActiveRecord or Mongoid)
-    # based on which Delayed::Backend has been loaded
+    # Load the correct module (ActiveRecord, Mongoid or Redis)
+    # based on which worker and backends are loaded
+    #
+    # Currently supports:
+    #  - Delayed Job with ActiveRecord and Mongoid
+    #  - Resque with Redis
     #
     # @return [nil]
     def self.included(base)
-      if defined?(Delayed::Backend::ActiveRecord::Job)
-        base.send(:include, ActiveRecord)
+
+      ##
+      # Delayed Job specific backends
+      if defined?(::Delayed::Job)
+        if defined?(::Delayed::Backend::ActiveRecord::Job)
+          base.send(:include, HireFire::Backend::DelayedJob::ActiveRecord)
+        end
+
+        if defined?(::Delayed::Backend::Mongoid::Job)
+          base.send(:include, HireFire::Backend::DelayedJob::Mongoid)
+        end
       end
 
-      if defined?(Delayed::Backend::Mongoid::Job)
-        base.send(:include, Mongoid)
+      ##
+      # Resque specific backends
+      if defined?(::Resque)
+        base.send(:include, HireFire::Backend::Resque::Redis)
       end
     end
 

data/lib/hirefire/backend/delayed_job/active_record.rb

@@ -0,0 +1,31 @@
+# encoding: utf-8
+
+module HireFire
+  module Backend
+    module DelayedJob
+      module ActiveRecord
+
+        ##
+        # Counts the amount of queued jobs in the database,
+        # failed jobs are excluded from the sum
+        #
+        # @return [Fixnum] the amount of pending jobs
+        def jobs
+          ::Delayed::Job.
+          where(:failed_at => nil).
+          where('run_at <= ?', Time.now).count
+        end
+
+        ##
+        # Counts the amount of jobs that are locked by a worker
+        #
+        # @return [Fixnum] the amount of (assumably working) workers
+        def workers
+          ::Delayed::Job.
+          where('locked_by IS NOT NULL').count
+        end
+
+      end
+    end
+  end
+end

data/lib/hirefire/backend/delayed_job/mongoid.rb

@@ -0,0 +1,32 @@
+# encoding: utf-8
+
+module HireFire
+  module Backend
+    module DelayedJob
+      module Mongoid
+
+        ##
+        # Counts the amount of queued jobs in the database,
+        # failed jobs and jobs scheduled for the future are excluded
+        #
+        # @return [Fixnum]
+        def jobs
+          ::Delayed::Job.where(
+            :failed_at  => nil,
+            :run_at.lte => Time.now
+          ).count
+        end
+
+        ##
+        # Counts the amount of jobs that are locked by a worker
+        #
+        # @return [Fixnum] the amount of (assumably working) workers
+        def workers
+          ::Delayed::Job.
+          where(:locked_by.ne => nil).count
+        end
+
+      end
+    end
+  end
+end

data/lib/hirefire/backend/resque/redis.rb

@@ -0,0 +1,20 @@
+# encoding: utf-8
+
+module HireFire
+  module Backend
+    module Resque
+      module Redis
+
+        ##
+        # Counts the amount of queued jobs in the database,
+        # failed jobs and jobs scheduled for the future are excluded
+        #
+        # @return [Fixnum]
+        def jobs
+          ::Resque.info[:pending].to_i + ::Resque.info[:working].to_i
+        end
+
+      end
+    end
+  end
+end

data/lib/hirefire/environment.rb

@@ -4,32 +4,56 @@ module HireFire
   module Environment
 
     ##
-    # This gets included in to the Delayed::Backend::(ActiveRecord|Mongoid)::Job
-    # classes and will add the necessary hooks (after_create, after_destroy and after_update)
-    # to spawn or kill Delayed Job worker processes on either Heroku or your local machine
+    # This module gets included in either:
+    #  - Delayed::Backend::ActiveRecord::Job
+    #  - Delayed::Backend::Mongoid::Job
+    #  - Resque::Job
+    #
+    # One of these classes will then be provided with an instance of one of the following:
+    #  - HireFire::Environment::Heroku
+    #  - HireFire::Environment::Local
+    #  - HireFire::Environment::Noop
+    #
+    # This instance is stored in the Class.environment class method
+    #
+    # The Delayed Job classes receive 3 hooks:
+    #  - hirefire_hire      ( invoked when a job gets queued )
+    #  - environment.fire   ( invoked when a queued job gets destroyed )
+    #  - environment.fire   ( invoked when a queued job gets updated unless the job didn't fail )
+    #
+    # The Resque classes get their hooks injected from the HireFire::Initializer#initialize! method
     #
     # @param (Class) base This is the class in which this module will be included
     def self.included(base)
-      base.send :extend, ClassMethods
-      base.class_eval do
-        after_create  'self.class.environment.hire'
-        after_destroy 'self.class.environment.fire'
-        after_update  'self.class.environment.fire',
-          :unless => Proc.new { |job| job.failed_at.nil? }
+      base.send :extend, HireFire::Environment::ClassMethods
+
+      ##
+      # Only implement these hooks for Delayed::Job backends
+      if base.name =~ /Delayed::Backend::(ActiveRecord|Mongoid)::Job/
+        base.send :extend, HireFire::Environment::DelayedJob::ClassMethods
+
+        base.class_eval do
+          after_create  'self.class.hirefire_hire'
+          after_destroy 'self.class.environment.fire'
+          after_update  'self.class.environment.fire',
+            :unless => Proc.new { |job| job.failed_at.nil? }
+        end
       end
 
       Logger.message("#{ base.name } detected!")
     end
 
     ##
-    # Class methods that will be added to the Delayed::Job backend
+    # Class methods that will be added to the
+    # Delayed::Job and Resque::Job classes
     module ClassMethods
 
       ##
-      # Returns the environment class method (for Delayed::Job ORM/ODM class)
+      # Returns the environment class method (containing an instance of the proper environment class)
+      # for either Delayed::Job or Resque::Job
       #
       # If HireFire.configuration.environment is nil (the default) then it'll
-      # auto-detect which environment to run in (either Heroku or Local)
+      # auto-detect which environment to run in (either Heroku or Noop)
       #
       # If HireFire.configuration.environment isn't nil (explicitly set) then
       # it'll run in the specified environment (Heroku, Local or Noop)
@@ -46,5 +70,34 @@ module HireFire
       end
     end
 
+    ##
+    # Delayed Job specific module
+    module DelayedJob
+      module ClassMethods
+
+        ##
+        # This method is an attempt to improve web-request throughput.
+        #
+        # A class method for Delayed::Job which first checks if any worker is currently
+        # running by checking to see if there are any jobs locked by a worker. If there aren't
+        # any jobs locked by a worker there is a high chance that there aren't any workers running.
+        # If this is the case, then we sure also invoke the 'self.class.environment.hire' method
+        #
+        # Another check is to see if there is only 1 job (which is the one that
+        # was just added before this callback invoked). If this is the case
+        # then it's very likely there aren't any workers running and we should
+        # invoke the 'self.class.environment.hire' method to make sure this is the case.
+        #
+        # @return [nil]
+        def hirefire_hire
+          delayed_job = ::Delayed::Job.new
+          if delayed_job.workers == 0 \
+          or delayed_job.jobs    == 1
+            environment.hire
+          end
+        end
+      end
+    end
+
   end
 end