samoli-hirefire 0.1.1

data/.document

@@ -0,0 +1,3 @@
+README.md
+LICENSE
+lib/**/*.rb

data/.gitignore

@@ -0,0 +1,4 @@
+.DS_Store
+*.gem
+.yardoc/*
+.yardoc/**/*

data/.infinity_test

@@ -0,0 +1,11 @@
+# encoding: utf-8
+
+##
+# Run all tests when a file gets saved inside the lib folder
+infinity_test do
+  heuristics do
+    add('lib/') do |file|
+      run :all => :tests
+    end
+  end
+end

data/.rspec

@@ -0,0 +1,3 @@
+--format Fuubar
+--color
+--tty

data/Gemfile

@@ -0,0 +1,15 @@
+# encoding: utf-8
+
+source 'http://rubygems.org'
+
+##
+# Define gems to be used in the 'test' environment
+group :test do
+  gem 'heroku'
+  gem 'rush'
+  gem 'rspec'
+  gem 'mocha'
+  gem 'infinity_test'
+  gem 'fuubar'
+  gem 'timecop'
+end

data/Gemfile.lock

@@ -0,0 +1,53 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    configuration (1.2.0)
+    diff-lcs (1.1.2)
+    fattr (2.2.0)
+    fuubar (0.0.3)
+      rspec (~> 2.0)
+      rspec-instafail (~> 0.1.4)
+      ruby-progressbar (~> 0.0.9)
+    heroku (1.20.1)
+      launchy (~> 0.3.2)
+      rest-client (< 1.7.0, >= 1.4.0)
+    infinity_test (1.0.2)
+      notifiers (>= 1.1.0)
+      watchr (>= 0.7)
+    launchy (0.3.7)
+      configuration (>= 0.0.5)
+      rake (>= 0.8.1)
+    mime-types (1.16)
+    mocha (0.9.12)
+    notifiers (1.1.0)
+    rake (0.8.7)
+    rest-client (1.6.1)
+      mime-types (>= 1.16)
+    rspec (2.5.0)
+      rspec-core (~> 2.5.0)
+      rspec-expectations (~> 2.5.0)
+      rspec-mocks (~> 2.5.0)
+    rspec-core (2.5.1)
+    rspec-expectations (2.5.0)
+      diff-lcs (~> 1.1.2)
+    rspec-instafail (0.1.6)
+    rspec-mocks (2.5.0)
+    ruby-progressbar (0.0.9)
+    rush (0.6.7)
+      session
+    session (3.1.0)
+      fattr
+    timecop (0.3.5)
+    watchr (0.7)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  fuubar
+  heroku
+  infinity_test
+  mocha
+  rspec
+  rush
+  timecop

data/LICENSE.md

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Michael van Rooijen ( [@meskyanichi](http://twitter.com/#!/meskyanichi) )
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,148 @@
+HireFire - The Heroku Worker Manager
+====================================
+
+**HireFire automatically "hires" and "fires" (aka "scales") [Delayed Job](https://github.com/collectiveidea/delayed_job) and [Resque](https://github.com/defunkt/resque) workers on Heroku**. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases.
+
+**Low traffic example** say we have a small application that doesn't process for more than 2 hours in the background a month. Meanwhile, your worker is basically just idle the rest of the 718 hours in that month. Keeping that idle worker running costs $36/month ($0.05/hour). But, for the resources you're actually **making use of** (2 hours a month), you should be paying $0.10/month, not $36/month. This is what HireFire is for.
+
+**High traffic example** say we have a high traffic application that needs to process a lot of jobs. There may be "traffic spikes" from time to time. In this case you can take advantage of the **job\_worker\_ratio**. Since this is application-specific, HireFire allows you to define how many workers there should be running, depending on the amount of queued jobs there are (see example configuration below). HireFire will then spin up more workers as traffic increases so it can work through the queue faster, then when the jobs are all finished, it'll shut down all the workers again until the next job gets queued (in which case it'll start with only a single worker again).
+
+**Enough with the examples!** Read on to see how to set it, and configure it to your scaling and money saving needs.
+
+Author
+------
+
+**Michael van Rooijen ( [@meskyanichi](http://twitter.com/#!/meskyanichi) )**
+
+Drop me a message for any questions, suggestions, requests, bugs or submit them to the [issue log](https://github.com/meskyanichi/hirefire/issues).
+
+
+Setting it up
+-------------
+
+A painless process. In a Ruby on Rails environment you would do something like this.
+
+**Rails.root/Gemfile**
+
+    gem 'rails'
+    # gem 'delayed_job' # uncomment this line if you use Delayed Job
+    # gem 'resque'      # uncomment this line if you use Resque
+    gem 'hirefire'
+
+**(The order is important: "Delayed Job" / "Resque" > HireFire)**
+
+Be sure to add the following Heroku environment variables so HireFire can manage your workers.
+
+    heroku config:add HIREFIRE_EMAIL=<your_email> HIREFIRE_PASSWORD=<your_password>
+
+These are the same email and password credentials you use to log in to the Heroku web interface to manage your workers.
+
+And that's it. Next time you deploy to [Heroku](http://heroku.com/) it'll automatically hire and fire your workers. Now, there are defaults, but I highly recommend you configure it since it only takes a few seconds. Create an initializer file:
+
+**Rails.root/config/initializers/hirefire.rb**
+
+    HireFire.configure do |config|
+      config.environment      = nil # default in production is :heroku. default in development is :noop
+      config.max_workers      = 5   # default is 1
+      config.min_workers      = 0   # default is 0
+      config.job_worker_ratio = [
+          { :jobs => 1,   :workers => 1 },
+          { :jobs => 15,  :workers => 2 },
+          { :jobs => 35,  :workers => 3 },
+          { :jobs => 60,  :workers => 4 },
+          { :jobs => 80,  :workers => 5 }
+        ]
+    end
+
+Basically what it comes down to is that we say **NEVER** to hire more than 5 workers at a time (`config.max_workers = 5`). And then we define an array of hashes that represents our **job\_worker\_ratio**. In the above example we are basically saying:
+
+* Hire 1 worker if there are 1-14 queued jobs
+* Hire 2 workers if there are 15-34 queued jobs
+* Hire 3 workers if there are 35-59 queued jobs
+* Hire 4 workers if there are 60-79 queued jobs
+* Hire 5 workers if there are more than 80 queued jobs
+
+Once all the jobs in the queue have been processed, it'll fire (shut down) all the workers and start with a single worker the next time a new job gets queued. And then the next time the queue hits 15 jobs mark, in which case the single worker isn't fast enough on it's own, it'll spin up the 2nd worker again.
+
+*If you prefer a more functional way of defining your job/worker ratio, you could use the following notation style:*
+
+    HireFire.configure do |config|
+      config.max_workers = 5
+      config.job_worker_ratio = [
+        { :when => lambda {|jobs| jobs < 15 }, :workers => 1 },
+        { :when => lambda {|jobs| jobs < 35 }, :workers => 2 },
+        { :when => lambda {|jobs| jobs < 60 }, :workers => 3 },
+        { :when => lambda {|jobs| jobs < 80 }, :workers => 4 }
+      ]
+    end
+
+The above notation is slightly different, since now you basically define how many workers to hire when `jobs < n`. So for example if there are 80 or more jobs, it'll hire the `max_workers` amount, which is `5` in the above example. If you change the `max_workers = 5` to `max_workers = 10`, then if there are 80 or more jobs queued, it'll go from 4 to 10 workers.
+
+
+In a non-Ruby on Rails environment
+----------------------------------
+
+Almost the same setup, except that you have to initialize HireFire yourself after Delayed Job or Resque is done loading.
+
+    require 'delayed_job'
+    # require 'delayed_job' # uncomment this line if you use Delayed Job
+    # require 'resque'      # uncomment this line if you use Resque
+    HireFire::Initializer.initialize!
+
+**(Again, the order is important: "Delayed Job" / "Resque" > HireFire)**
+
+If all goes well you should see a message similar to this when you boot your application:
+
+    [HireFire] Delayed::Backend::ActiveRecord::Job detected!
+
+
+Worker / Mapper Support
+--------------
+
+HireFire currently works with the following worker and mapper libraries:
+
+- [Delayed Job](https://github.com/collectiveidea/delayed_job)
+  - [ActiveRecord ORM](https://github.com/rails/rails/tree/master/activerecord)
+  - [Mongoid ODM](https://github.com/mongoid/mongoid) (using [delayed_job_mongoid](https://github.com/collectiveidea/delayed_job_mongoid))
+
+- [Resque](https://github.com/defunkt/resque)
+  - [Redis](https://github.com/ezmobius/redis-rb)
+
+
+Frequently Asked Questions
+--------------------------
+
+- **Question:** *Does it start workers immediately after a job gets queued?*
+  - **Answer:** Yes, once a new job gets queued it'll immediately calculate the amount of workers that are required and hire them accordingly.
+
+- **Question:** *Does it stop workers immediately when there are no jobs to be processed?*
+  - **Answer:** Yes, every worker has been made self-aware to see this. Once there are no jobs to be processed, all workers will immediately be fired (shut down). *For example, if you have no jobs in the queue, and you start cranking up your Workers via Heroku's web ui, once the worker spawns and sees it has nothing to do, it'll immediately shut itself down.*
+
+- **Question:** *How does this save me money?*
+  - **Answer:** According to Heroku's documentation, Workers (same as Dynos), are prorated to the second. *For example, say that 10 jobs get queued and a worker is spawned to process them and takes about 1 minute to do so and then shuts itself down, theoretically you only pay $0.0008.*
+
+- **Question:** *With Delayed Job you can set the :run_at to a time in the future.*
+  - **Answer:** Unfortunately since we cannot spawn a monitoring process on the Heroku platform, HireFire will not hire workers until a job gets queued. This means that if you set the :run_at time a few minutes in the future, and these few minutes pass, the job will not be processed until a new job gets queued which triggers the chain of events. (Best to avoid using `run_at` with Delayed Job when using HireFire unless you have a mid-high traffic web application in which cause HireFire gets triggered enough times)
+
+- **Question:** *If a job is set to run at a time in the future, will workers remain hired to wait for this job to be "processable"?*
+  - **Answer:** No, because if you enqueue a job to run 3 hours from the time it was enqueued, you might have workers doing nothing the coming 3 hours. Best to avoid scheduling jobs to be processed in the future.
+
+- **Question:** *Will it scale down workers from, for example, 5 to 4?*
+  - **Answer:** No, I have consciously chosen not to do that for 2 reasons:
+      1. There is no way to tell which worker is currently processing a job, so it might fire a worker that was busy, causing the job to be exit during the process.
+      2. Does it really matter? Your jobs will be processed faster, and once the queue is completely empty, all workers will be fire anyway. (You could call this a feature! Since 5 jobs process faster than 4, but the cost remains the same cause it's all pro-rated to the second)
+
+- **Question:** *Will running jobs concurrently (with multiple Worker) cost more?*
+  - **Answer:** Actually, no. Since worker's are pro-rated to the second, the moment you hire 3 workers, it costs 3 times more, but it also processes 3 times faster. You could also let 1 worker process all the jobs rather than 3, but that means it'll still cost the same amount as when you hire 3 workers, since it takes 3 times longer to process.
+
+- **Question:** *Can I process jobs faster with HireFire?*
+  - **Answer:** When you run multiple jobs concurrently, you can speed up your processing dramatically. *Normally you wouldn't set the workers to 10 for example, but with HireFire you can tell it to Hire 10 workers when there are 50 jobs (would normally be overkill and cost you A LOT of money) but since (see Q/A above) Workers are pro-rated to the second, and HireFire immediately fires all workers once all the jobs in the queue have been processed, it makes no different whether you have a single worker processing 50 jobs, or 5 workers, or even 10 workers. It processes 10 times faster, but costs the same.*
+
+
+
+Other potentially interesting gems
+----------------------------------
+
+* [Backup](https://github.com/meskyanichi/backup)
+* [GitPusshuTen](https://github.com/meskyanichi/gitpusshuten)
+* [Mongoid::Paperclip](https://github.com/meskyanichi/mongoid-paperclip)

data/hirefire.gemspec

@@ -0,0 +1,29 @@
+# encoding: utf-8
+
+require File.expand_path(File.dirname(__FILE__) + '/lib/hirefire')
+
+Gem::Specification.new do |gem|
+
+  ##
+  # General configuration / information
+  gem.name        = 'samoli-hirefire'
+  gem.version     = HireFire::Version.current
+  gem.platform    = Gem::Platform::RUBY
+  gem.authors     = 'Michael van Rooijen'
+  gem.email       = 'meskyanichi@gmail.com'
+  gem.homepage    = 'http://rubygems.org/gems/hirefire'
+  gem.summary     = %|HireFire automatically "hires" and "fires" (aka "scales") Delayed Job and Resque workers on Heroku.|
+  gem.description = %|HireFire automatically "hires" and "fires" (aka "scales") Delayed Job and Resque workers on Heroku. When there are no queue jobs, HireFire will fire (shut down) all workers. If there are queued jobs, then it'll hire (spin up) workers. The amount of workers that get hired depends on the amount of queued jobs (the ratio can be configured by you). HireFire is great for both high, mid and low traffic applications. It can save you a lot of money by only hiring workers when there are pending jobs, and then firing them again once all the jobs have been processed. It's also capable to dramatically reducing processing time by automatically hiring more workers when the queue size increases.|
+
+  ##
+  # Files and folder that need to be compiled in to the Ruby Gem
+  gem.files         = %x[git ls-files].split("\n")
+  gem.test_files    = %x[git ls-files -- {spec}/*].split("\n")
+  gem.require_path  = 'lib'
+
+  ##
+  # Production gem dependencies
+  gem.add_dependency 'heroku', ['~> 1.20.1']
+  gem.add_dependency 'rush',   ['~> 0.6.7']
+
+end

data/lib/hirefire.rb

@@ -0,0 +1,111 @@
+# encoding: utf-8
+
+module HireFire
+
+  ##
+  # HireFire constants
+  LIB_PATH         = File.dirname(__FILE__)
+  HIREFIRE_PATH    = File.join(LIB_PATH,      'hirefire')
+  ENVIRONMENT_PATH = File.join(HIREFIRE_PATH, 'environment')
+  BACKEND_PATH     = File.join(HIREFIRE_PATH, 'backend')
+  WORKERS_PATH     = File.join(HIREFIRE_PATH, 'workers')
+
+  ##
+  # HireFire namespace
+  autoload :Configuration, File.join(HIREFIRE_PATH, 'configuration')
+  autoload :Environment,   File.join(HIREFIRE_PATH, 'environment')
+  autoload :Initializer,   File.join(HIREFIRE_PATH, 'initializer')
+  autoload :Backend,       File.join(HIREFIRE_PATH, 'backend')
+  autoload :Logger,        File.join(HIREFIRE_PATH, 'logger')
+  autoload :Version,       File.join(HIREFIRE_PATH, 'version')
+
+  ##
+  # HireFire::Environment namespace
+  module Environment
+    autoload :Base,   File.join(ENVIRONMENT_PATH, 'base')
+    autoload :Heroku, File.join(ENVIRONMENT_PATH, 'heroku')
+    autoload :Local,  File.join(ENVIRONMENT_PATH, 'local')
+    autoload :Noop,   File.join(ENVIRONMENT_PATH, 'noop')
+  end
+
+  ##
+  # HireFire::Workers namespace
+  module Workers
+    autoload :DelayedJob, File.join(WORKERS_PATH, 'delayed_job')
+    autoload :Resque,     File.join(WORKERS_PATH, 'resque')
+  end
+
+  ##
+  # HireFire::Backend namespace
+  module Backend
+    DELAYED_JOB_PATH = File.join(BACKEND_PATH, 'delayed_job')
+    RESQUE_PATH      = File.join(BACKEND_PATH, 'resque')
+
+    ##
+    # HireFire::Backend::DelayedJob namespace
+    module DelayedJob
+      autoload :ActiveRecord, File.join(DELAYED_JOB_PATH, 'active_record')
+      autoload :Mongoid,      File.join(DELAYED_JOB_PATH, 'mongoid')
+    end
+
+    ##
+    # HireFire::Backend::Resque namespace
+    module Resque
+      autoload :Redis, File.join(RESQUE_PATH, 'redis')
+    end
+  end
+
+  ##
+  # This method is used to configure HireFire
+  #
+  # @yield [config] the instance of HireFire::Configuration class
+  # @yieldparam [Fixnum] max_workers default: 1 (set at least 1)
+  # @yieldparam [Array] job_worker_ratio default: see example
+  # @yieldparam [Symbol, nil] environment (:heroku, :local, :noop or nil) - default: nil
+  #
+  # @note Every param has it's own defaults. It's best to leave the environment param at "nil".
+  #   When environment is set to "nil", it'll default to the :noop environment. This basically means
+  #   that you have to run "rake jobs:work" yourself from the console to get the jobs running in development mode.
+  #   In production, it'll automatically use :heroku if deployed to the Heroku platform.
+  #
+  # @example
+  #   HireFire.configure do |config|
+  #     config.environment      = nil
+  #     config.max_workers      = 5
+  #     config.job_worker_ratio = [
+  #       { :jobs => 1,   :workers => 1 },
+  #       { :jobs => 15,  :workers => 2 },
+  #       { :jobs => 35,  :workers => 3 },
+  #       { :jobs => 60,  :workers => 4 },
+  #       { :jobs => 80,  :workers => 5 }
+  #     ]
+  #   end
+  #
+  # @return [nil]
+  def self.configure
+    yield(configuration); nil
+  end
+
+  ##
+  # Instantiates a new HireFire::Configuration
+  # instance and instance variable caches it
+  def self.configuration
+    @configuration ||= HireFire::Configuration.new
+  end
+
+end
+
+##
+# If Ruby on Rails is detected, it'll automatically initialize HireFire
+# so that the developer doesn't have to manually invoke it from an initializer file
+#
+# Users not using Ruby on Rails will have to run "HireFire::Initializer.initialize!"
+# in their application manually, after loading the worker library (either "Delayed Job" or "Resque")
+# and the desired mapper (ActiveRecord, Mongoid or Redis)
+if defined?(Rails)
+  if Rails.version >= '3.0.0'
+    require File.join(HireFire::HIREFIRE_PATH, 'railtie')
+  else
+    HireFire::Initializer.initialize!
+  end
+end

data/lib/hirefire/backend.rb

@@ -0,0 +1,37 @@
+# encoding: utf-8
+
+module HireFire
+  module Backend
+
+    ##
+    # Load the correct module (ActiveRecord, Mongoid or Redis)
+    # based on which worker and backends are loaded
+    #
+    # Currently supports:
+    #  - Delayed Job with ActiveRecord and Mongoid
+    #  - Resque with Redis
+    #
+    # @return [nil]
+    def self.included(base)
+
+      ##
+      # Delayed Job specific backends
+      if defined?(::Delayed::Job)
+        if defined?(::Delayed::Backend::ActiveRecord::Job)
+          base.send(:include, HireFire::Backend::DelayedJob::ActiveRecord)
+        end
+
+        if defined?(::Delayed::Backend::Mongoid::Job)
+          base.send(:include, HireFire::Backend::DelayedJob::Mongoid)
+        end
+      end
+
+      ##
+      # Resque specific backends
+      if defined?(::Resque)
+        base.send(:include, HireFire::Backend::Resque::Redis)
+      end
+    end
+
+  end
+end