workless_revived 1.2.4

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 90ea3d00f5b27151452d15ba4e6f2a07785c92dc
+  data.tar.gz: de0e6eecd3fcfffbf9615035d892d0345b1a3107
+SHA512:
+  metadata.gz: cd5bbde2310761f1064b5f2e9ef3544c710e198ef496f7322468b643e59ccc297bfa9188da596d392c32fefc3ed916724db3a523525c4d332fd5a7e6fff744a2
+  data.tar.gz: 078ef22d2b35c911ae87477748f8794f4de6eef47e0ff8cf8d7d911939a206e808bebfada3e21b9d8b70ec2fc511ee3edd012e61f6dc33c29b9158c3e8dd9869

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 lostboy
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,129 @@
+[![Build Status](https://secure.travis-ci.org/davidakachaos/workless_revived.png?branch=master)](http://travis-ci.org/davidakachaos/workless_revived)
+[![Gem Version](https://badge.fury.io/rb/workless_revived.png)](http://badge.fury.io/rb/workless_revived)
+[![Test Coverage](https://coveralls.io/repos/davidakachaos/workless_revived/badge.png?branch=master)](https://coveralls.io/r/davidakachaos/workless_revived)
+
+# Workless Revived
+
+This gem was originally created by [lostboy](https://github.com/lostboy). Since he became inactive, I've decided to revive this project under a new(ish) name.
+
+This is an addon for delayed_job (> 2.0.0) http://github.com/collectiveidea/delayed_job
+It is designed to be used when you're using Heroku as a host and have the need to do background work with delayed job but you don't want to leave the workers running all the time as it costs money.
+
+By adding the gem to your project and configuring our Heroku app with some config variables workless should do the rest.
+
+## Updates
+
+* Version 1.2.4 drops support for older versions!
+* Version 1.2.3 replaces multiple commit callback with two callbacks for compatibility by @lostboy
+* Version 1.2.2 includes after_commit fix by @collectiveip
+* Version 1.2.1 includes support for Rails 4 & DJ 4 by @florentmorin
+* Version 1.2.0 includes new support for Sequel by @davidakachaos
+* Version 1.1.3 includes changes by @radanskoric to reduce number of heroku api calls
+* Version 1.1.2 includes a change by @davidakachaos to scale workers using after_commit
+* Version 1.1.1 includes a fix from @filiptepper and @fixr to correctly scale workers
+* Version 1.1.0 has been released, this adds support for scaling using multiple workers thanks to @jaimeiniesta and @davidakachaos.
+* Version 1.0.0 has been released, this brings compatibility with delayed_job 3 and compatibility with Rails 2.3.x and up.
+
+## Compatibility
+
+Workless should work correctly with Rubies 2.0.0 and up. It is compatible with Delayed Job since version 2.0.7 up to the latest version 4.1.2, the table below shows tested compatibility with ruby, rails and delayed_job
+
+Ruby | Rails  | Delayed Job
+---------- | ------ | -----
+2.0.0  | 3.2.22 | 2.0.7
+2.2.4      | 4.2    | 2.1.4
+2.3.0      | 5.0    | 4.1.2
+
+## Installation
+
+Add the workless gem and the delayed_job gem to your project Gemfile and update your bundle. Its is recommended to specify the gem version for delayed_job
+
+### For rails 3.x with latest delayed_job 3.x using active record
+
+<pre>
+gem "delayed_job_active_record"
+gem "workless", "~> 1.1.3"
+</pre>
+
+### For rails 4.x with latest delayed_job 3.x using active record
+
+<pre>
+gem "delayed_job_active_record"
+gem "workless_revived", "~> 1.2.4"
+</pre>
+
+### For rails 5.x with latest delayed_job 3.x using active record
+
+<pre>
+gem "delayed_job_active_record"
+gem "workless_revived", "~> 1.2.4"
+</pre>
+
+
+If you don't specify delayed_job in your Gemfile workless will bring it in, most likely the latest version (4.1.2)
+
+Add your Heroku app name / [API key](https://devcenter.heroku.com/articles/authentication) as config vars to your Heroku instance.
+
+<pre>
+heroku config:add HEROKU_API_KEY=yourapikey APP_NAME=yourherokuappname
+</pre>
+
+## Failing Jobs
+
+In the case of failed jobs Workless will only shut down the dj worker if all attempts have been tried. By default Delayed Job will try 25 times to process a job with ever increasing time delays between each unsucessful attempt. Because of this Workless configures Delayed Job to try failed jobs only 3 times to reduce the amount of time a worker can be running while trying to process them.
+
+## Configuration
+
+Workless can be disabled by using the null scaler that will ignore the workers requests to scale up and down. In an environment file add this in the config block:
+
+<pre>
+config.after_initialize do
+  Delayed::Job.scaler = :null
+end
+</pre>
+
+There are three other scalers included. Note that if you are running on the Aspen or Bamboo stacks on Heroku and you don't explicitly specify the scaler, the heroku scaler will be used automatically.
+
+<pre>
+Delayed::Job.scaler = :heroku
+Delayed::Job.scaler = :heroku_cedar
+Delayed::Job.scaler = :local
+</pre>
+
+The local scaler uses @adamwiggins rush library http://github.com/adamwiggins/rush to start and stop workers on a local machine. The local scaler also relies on script/delayed_job (which in turn requires the daemon gem). If you have been using foreman to run your workers, go back and see the delayed_job [setup instructions](https://github.com/collectiveidea/delayed_job/blob/master/README.md).
+
+The heroku scaler works on the Aspen and Bamboo stacks while the heroku_cedar scaler only works on the new Cedar stack.
+
+## Scaling to multiple workers
+
+As an experimental feature for the Cedar stack, Workless can scale to more than 1 worker based on the current work load. You just need to define these config variables on your app, setting the values you want:
+
+<pre>
+heroku config:add WORKLESS_MAX_WORKERS=10
+heroku config:add WORKLESS_MIN_WORKERS=0
+heroku config:add WORKLESS_WORKERS_RATIO=50
+</pre>
+
+In this example, it will scale up to a maximum of 10 workers, firing up 1 worker for every 50 jobs on the queue. The minimum will be 0 workers, but you could set it to a higher value if you want.
+
+## How does Workless work?
+
+- `Delayed::Workless::Scaler` is mixed into the `Delayed::Job` class, which adds a bunch of callbacks to it.
+- When a job is created on the database, a `create` callback starts a worker.
+- The worker runs the job, which removes it from the database.
+- A `destroy` callback stops the worker.
+
+## Note on Patches/Pull Requests
+
+* Please fork the project.
+* Make your feature addition or bug fix.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request.
+
+## Copyright
+
+Copyright (c) 2010 lostboy.
+Copyright (c) 2016 davidakachaos.
+
+See LICENSE for details.

data/lib/workless.rb

@@ -0,0 +1,5 @@
+require "heroku-api"
+require File.dirname(__FILE__) + "/workless/scalers/base"
+require File.dirname(__FILE__) + "/workless/scaler"
+require File.dirname(__FILE__) + "/workless/middleware/workless_checker" if defined?(Rails::Railtie)
+require File.dirname(__FILE__) + "/workless/railtie" if defined?(Rails::Railtie)

data/lib/workless/initialize.rb

@@ -0,0 +1,5 @@
+Delayed::Worker.max_attempts = 3
+Delayed::Backend::ActiveRecord::Job.send(:include, Delayed::Workless::Scaler) if defined?(Delayed::Backend::ActiveRecord::Job)
+Delayed::Backend::Mongoid::Job.send(:include, Delayed::Workless::Scaler) if defined?(Delayed::Backend::Mongoid::Job)
+Delayed::Backend::MongoMapper::Job.send(:include, Delayed::Workless::Scaler) if defined?(Delayed::Backend::MongoMapper::Job)
+Delayed::Backend::Sequel::Job.send(:include, Delayed::Workless::Scaler) if defined?(Delayed::Backend::Sequel::Job)

data/lib/workless/middleware/workless_checker.rb

@@ -0,0 +1,37 @@
+class WorklessChecker
+  def initialize(app)
+    @app = app
+  end
+
+  def call(env)
+    status, headers, response = @app.call(env)
+    return [status, headers, response] if file?(headers) || empty?(response)
+
+    Delayed::Job.scaler.up if Delayed::Job.scaler.jobs.size > 0
+    response_body = nil
+    if status == 200 && !response.body.frozen? && html_request?(headers, response)
+      response_body = response.body << "\n<!-- workless jobs: #{Delayed::Job.scaler.jobs.size} -->"
+      headers['Content-Length'] = response_body.bytesize.to_s
+    end
+
+    return [status, headers, response_body ? [response_body] : response]
+  end
+
+  # fix issue if response's body is a Proc
+  def empty?(response)
+    # response may be ["Not Found"], ["Move Permanently"], etc.
+    (response.is_a?(Array) && response.size <= 1) ||
+      !response.respond_to?(:body) ||
+      !response.body.respond_to?(:empty?) ||
+      response.body.empty?
+  end
+
+  # if send file?
+  def file?(headers)
+    headers["Content-Transfer-Encoding"] == "binary"
+  end
+
+  def html_request?(headers, response)
+    headers['Content-Type'] && headers['Content-Type'].include?('text/html') && response.body.include?("<html")
+  end
+end

data/lib/workless/railtie.rb

@@ -0,0 +1,11 @@
+require 'rails'
+require 'delayed_job'
+
+module Delayed
+  class Railtie < Rails::Railtie
+    initializer :after_initialize do |config|
+      require 'workless/initialize'
+      config.middleware.use "WorklessChecker"
+    end
+  end
+end

data/lib/workless/scaler.rb

@@ -0,0 +1,57 @@
+module Delayed
+  module Workless
+    module Scaler
+
+      autoload :Heroku,      "workless/scalers/heroku"
+      autoload :HerokuCedar, "workless/scalers/heroku_cedar"
+      autoload :Local,       "workless/scalers/local"
+      autoload :Null,        "workless/scalers/null"
+
+      def self.included(base)
+        base.send :extend, ClassMethods
+        if base.to_s =~ /ActiveRecord/
+          base.class_eval do
+            after_commit "self.class.scaler.down", :on => :update, :if => Proc.new {|r| !r.failed_at.nil? }
+            after_commit "self.class.scaler.down", :on => :destroy, :if => Proc.new {|r| r.destroyed? or !r.failed_at.nil? }
+            after_commit "self.class.scaler.up", :on => :create
+          end          
+        elsif base.to_s =~ /Sequel/
+          base.send(:define_method, 'after_destroy') do
+            super
+            self.class.scaler.down
+          end
+          base.send(:define_method, 'after_create') do
+            super
+            self.class.scaler.up
+          end
+          base.send(:define_method, 'after_update') do
+            super
+            self.class.scaler.down
+          end
+        else
+          base.class_eval do
+            after_destroy "self.class.scaler.down"
+            after_create "self.class.scaler.up"
+            after_update "self.class.scaler.down", :unless => Proc.new {|r| r.failed_at.nil? }
+          end
+        end
+      end
+
+      module ClassMethods
+        def scaler
+          @scaler ||= if ENV.include?("HEROKU_API_KEY")
+            Scaler::HerokuCedar
+          else
+            Scaler::Local
+          end
+        end
+
+        def scaler=(scaler)
+          @scaler = "Delayed::Workless::Scaler::#{scaler.to_s.camelize}".constantize
+        end
+      end
+
+    end
+
+  end
+end

data/lib/workless/scalers/base.rb

@@ -0,0 +1,27 @@
+require 'delayed_job'
+
+module Delayed
+  module Workless
+    module Scaler
+
+      class Base
+        def self.jobs
+          if Rails.version >= "3.0.0"
+            Delayed::Job.where(:failed_at => nil)
+          else
+            Delayed::Job.all(:conditions => { :failed_at => nil })
+          end
+        end
+      end
+
+      module HerokuClient
+
+        def client
+          @client ||= ::Heroku::API.new(:api_key => ENV['HEROKU_API_KEY'])
+        end
+
+      end
+
+    end
+  end
+end

data/lib/workless/scalers/heroku.rb

@@ -0,0 +1,27 @@
+require 'heroku-api'
+
+module Delayed
+  module Workless
+    module Scaler
+
+      class Heroku < Base
+
+        extend Delayed::Workless::Scaler::HerokuClient
+
+        def self.up
+          client.put_workers(ENV['APP_NAME'], 1) if self.workers == 0
+        end
+
+        def self.down
+          client.put_workers(ENV['APP_NAME'], 0) unless self.jobs.count > 0 or self.workers == 0
+        end
+
+        def self.workers
+          client.get_ps(ENV['APP_NAME']).body.count { |p| p["process"] =~ /worker\.\d?/ }
+        end
+
+      end
+
+    end
+  end
+end

data/lib/workless/scalers/heroku_cedar.rb

@@ -0,0 +1,49 @@
+require 'heroku-api'
+
+module Delayed
+  module Workless
+    module Scaler
+      class HerokuCedar < Base
+        extend Delayed::Workless::Scaler::HerokuClient
+
+        def self.up
+          client.post_ps_scale(ENV['APP_NAME'], 'worker', self.workers_needed) if self.workers_needed > self.min_workers and self.workers < self.workers_needed
+        end
+
+        def self.down
+          client.post_ps_scale(ENV['APP_NAME'], 'worker', self.min_workers) unless self.jobs.count > 0 or self.workers == self.min_workers
+        end
+
+        def self.workers
+          client.get_ps(ENV['APP_NAME']).body.count { |p| p["process"] =~ /worker\.\d?/ }
+        end
+
+        # Returns the number of workers needed based on the current number of pending jobs and the settings defined by:
+        #
+        # ENV['WORKLESS_WORKERS_RATIO']
+        # ENV['WORKLESS_MAX_WORKERS']
+        # ENV['WORKLESS_MIN_WORKERS']
+        #
+        def self.workers_needed
+          [[(self.jobs.count.to_f / self.workers_ratio).ceil, self.max_workers].min, self.min_workers].max
+        end
+
+        def self.workers_ratio
+          if ENV['WORKLESS_WORKERS_RATIO'].present? && (ENV['WORKLESS_WORKERS_RATIO'].to_i != 0)
+            ENV['WORKLESS_WORKERS_RATIO'].to_i
+          else
+            100
+          end
+        end
+
+        def self.max_workers
+          ENV['WORKLESS_MAX_WORKERS'].present? ? ENV['WORKLESS_MAX_WORKERS'].to_i : 1
+        end
+
+        def self.min_workers
+          ENV['WORKLESS_MIN_WORKERS'].present? ? ENV['WORKLESS_MIN_WORKERS'].to_i : 0
+        end
+      end
+    end
+  end
+end

data/lib/workless/scalers/local.rb

@@ -0,0 +1,37 @@
+require 'rush'
+
+module Delayed
+  module Workless
+    module Scaler
+      class Local < Base
+
+        def self.executable_prefix
+          if defined? Delayed::Compatibility.executable_prefix
+            Delayed::Compatibility.executable_prefix
+          else
+            'script'
+          end
+        end
+
+        def self.up
+          if self.workers == 0
+            Rush::Box.new[Rails.root].bash("#{executable_prefix}/delayed_job start -i workless", :background => true)
+            sleep 1
+          end
+          true
+        end
+
+        def self.down
+          if self.workers > 0 and jobs.count == 0
+            Rush::Box.new[Rails.root].bash("#{executable_prefix}/delayed_job stop -i workless", :background => true)
+          end
+          true
+        end
+
+        def self.workers
+          Rush::Box.new.processes.filter(:cmdline => /delayed_job start -i workless|delayed_job.workless/).size
+        end
+      end
+    end
+  end
+end

data/lib/workless/scalers/null.rb

@@ -0,0 +1,17 @@
+module Delayed
+  module Workless
+    module Scaler
+
+      class Null < Base
+      
+        def self.up
+        end
+
+        def self.down
+        end
+
+      end
+      
+    end
+  end
+end

data/rails/init.rb

@@ -0,0 +1 @@
+require File.dirname(__FILE__) + "/../lib/workless/initialize"

metadata

@@ -0,0 +1,125 @@
+--- !ruby/object:Gem::Specification
+name: workless_revived
+version: !ruby/object:Gem::Version
+  version: 1.2.4
+platform: ruby
+authors:
+- davidakachaos
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-08-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 5.0.0.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 5.0.0.1
+- !ruby/object:Gem::Dependency
+  name: heroku-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rush
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: delayed_job
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.7
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.7
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Extension to Delayed Job to enable workers to scale up when needed
+email: davidakachaos@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- lib/workless.rb
+- lib/workless/initialize.rb
+- lib/workless/middleware/workless_checker.rb
+- lib/workless/railtie.rb
+- lib/workless/scaler.rb
+- lib/workless/scalers/base.rb
+- lib/workless/scalers/heroku.rb
+- lib/workless/scalers/heroku_cedar.rb
+- lib/workless/scalers/local.rb
+- lib/workless/scalers/null.rb
+- rails/init.rb
+homepage: http://github.com/davidakachaos/workless_revived
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 1.3.6
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5.1
+signing_key: 
+specification_version: 4
+summary: Use delayed job workers only when theyre needed on Heroku
+test_files: []