autoscale 0.9.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 116541fd00550df7c896d491b0861293d2bd6766
+  data.tar.gz: d7498ed8d5eeb0e2c483c7b70a92b575a6c7cea3
+SHA512:
+  metadata.gz: 7d9085be6c0eb40885b296cbe8d0a307f0a244b6278eb54f5bf366be865dfd8bb22da8a335d20a6802d90bf0b78115d2c6c627994973f5a737372fc8d877c9ee
+  data.tar.gz: d15705c79a8966dbf7439166fb3752dd5470298d90e236692a6c569f7ce9031b0214a477c9cfcb2d9fa7433d3315bf6c2268c4f07a064f52b964fd7277391c1f

data/CHANGELOG.md

@@ -0,0 +1,81 @@
+# Changelog
+
+## 0.10.0
+- Require Sidekiq 3.
+- Linear Scaling Strategy will not scale down past number of active workers. Assumes 1-1 SK process/dyno mapping.
+- QueueSystem#workers returns the number of engaged SK processes.
+
+## 0.9.0
+
+- CounterCacheRedis.new now takes a third parameter `worker_type`, a string used in the
+  Redis cache key. Allows for caching counts for various types of workers, not just `worker`
+- Support for Sidekiq 3.0
+- Strategy wrapper to ignore scheduled and retrying queues. Usage:
+    ``new_strategy = IgnoreScheduledAndRetrying.new(my_old_strategy)``
+- LinearScalingStrategy now accepts a minimum amount of work (as a percentage of worker capacity)
+  required to begin scaling up. E.g LinearScalingStrategy.new(10, 4, 0.5) will scale to one worker
+  after 4*0.5 = 2 jobs are enqueued, and a maximum of 10 workers at 10*4 jobs. Old behavior is preserved
+  with a default value of 0.
+
+## 0.8.0
+
+- Extracted caching of Heroku worker counts and added experimental Redis cache:
+    ``scaler.counter_cache = Autoscaler::CounterCacheRedis.new(Sidekiq.method(:redis))``
+- Now rescues Heroku::Api::Errors in addition to Excon::Errors
+
+## 0.7.0
+
+- Added Autoscaler::LinearScalingStrategy
+- EntireQueuSystem#queued always returns an integer
+
+## 0.6.0
+
+- Excon errors from the Heroku API are caught be default.  See `HerokuScaler#exception_handler` to override behavior
+- Client side scaling occurs after enquing the job, previously it was before.
+
+## 0.5.0
+
+- Experimental: `Client#set_initial_workers` to start workers on main process startup; typically:
+    Autoscaler::Sidekiq::Client.add_to_chain(chain, 'default' => heroku).set_initial_workers
+- Ensure that timeout is documented as being in seconds
+- Convert gemspec to wildcard file selection
+
+## 0.4.1
+
+- Missing file from gemspec
+
+## 0.4.0
+
+- Experimental: The default scaling logic is contained in BinaryScalingStrategy.  A strategy object can be passed instead of timeout to the server middleware.
+
+## 0.3.0
+
+- Downscale method changed from busy-waiting workers to a separate monitor process
+- Minimum Sidekiq version raised to 2.7 to take advantage of Worker API
+- Internal refactoring
+- Autoscaler::StubScaler may be used for local testing
+
+## 0.2.1
+
+- Separate background activity flags to avoid crosstalk between processes
+
+## 0.2.0
+
+- Raise minimum Sidekiq version to 2.6.1 to take advantage of Stats API
+- Inspect scheduled and retry sets to see if they match `specified_queues`
+- Testing: Refactor server middleware tests
+
+## 0.1.0
+
+- The `retry` and `scheduled` queues are now considered for shutdown
+- Testing: Guard starts up an isolated redis instance
+
+## 0.0.3
+
+- Typo correction
+
+## 0.0.2
+
+- Loosen Sidekiq version dependency
+- Add changelog
+- Add changelog, readme, and examples to gem files list

data/Guardfile

@@ -0,0 +1,12 @@
+guard 'process', :name => 'redis', :command => 'redis-server spec/redis_test.conf' do
+  watch('spec/redis_test.conf')
+end
+
+tag     = "--tag #{ENV['TAG']}"           if ENV['TAG']
+example = "--example '#{ENV['EXAMPLE']}'" if ENV['EXAMPLE']
+guard :rspec, :cmd => "rspec --color --format d #{tag} #{example}" do
+  watch(%r{^spec/.+_spec\.rb$})
+  watch(%r{^lib/(.+).rb$}) { |m| "spec/#{m[1]}_spec.rb" }
+  watch('spec/spec_helper.rb')  { "spec" }
+end
+

data/README.md

@@ -0,0 +1,81 @@
+# Sidekiq Heroku Autoscaler
+
+[Sidekiq](https://github.com/mperham/sidekiq) performs background jobs.  While its threading model allows it to scale easier than worker-pre-process background systems, people running test or lightly loaded systems on [Heroku](http://www.heroku.com/) still want to scale down to zero to avoid racking up charges.
+
+## Requirements
+
+Tested on Ruby 1.9.2 and Heroku Cedar stack.
+
+## Installation
+
+    gem install autoscaler
+
+## Getting Started
+
+This gem uses the [Heroku-Api](https://github.com/heroku/heroku.rb) gem, which requires an API key from Heroku.  It will also need the heroku app name.  By default, these are specified through environment variables.  You can also pass them to HerokuScaler explicitly.
+
+    HEROKU_API_KEY=.....
+    HEROKU_APP=....
+
+Install the middleware in your `Sidekiq.configure_` blocks
+
+    require 'autoscaler/sidekiq'
+    require 'autoscaler/heroku_scaler'
+
+    Sidekiq.configure_client do |config|
+      config.client_middleware do |chain|
+        chain.add Autoscaler::Sidekiq::Client, 'default' => Autoscaler::HerokuScaler.new
+      end
+    end
+
+    Sidekiq.configure_server do |config|
+      config.server_middleware do |chain|
+        chain.add(Autoscaler::Sidekiq::Server, Autoscaler::HerokuScaler.new, 60) # 60 second timeout
+      end
+    end
+
+## Limits and Challenges
+
+- HerokuScaler includes an attempt at current-worker cache that may be overcomplication, and doesn't work very well on the server
+- Multiple scale-down loops may be started, particularly if there are multiple jobs queued when the servers comes up.  Heroku seems to handle multiple scale-down commands well.
+- The scale-down monitor is triggered on job completion (and server middleware is only run around jobs), so if the server nevers processes any jobs, it won't turn off.
+- The retry and schedule lists are considered - if you schedule a long-running task, the process will not scale-down.
+- If background jobs trigger jobs in other scaled processes, please note you'll need `config.client_middleware` in your `Sidekiq.configure_server` block in order to scale-up.
+- Exceptions while calling the Heroku API are caught and printed by default.  See `HerokuScaler#exception_handler` to override
+
+## Experimental
+
+### Strategies
+
+You can pass a scaling strategy object instead of the timeout to the server middleware.  The object (or lambda) should respond to `#call(system, idle_time)` and return the desired number of workers.  See `lib/autoscaler/binary_scaling_strategy.rb` for an example.
+
+### Initial Workers
+
+`Client#set_initial_workers` to start workers on main process startup; typically:
+
+    Autoscaler::Sidekiq::Client.add_to_chain(chain, 'default' => heroku).set_initial_workers
+
+### Working caching
+
+    scaler.counter_cache = Autoscaler::CounterCacheRedis(Sidekiq.method(:redis))
+
+## Tests
+
+The project is setup to run RSpec with Guard.  It expects a redis instance on a custom port, which is started by the Guardfile.
+
+The HerokuScaler is not tested by default because it makes live API requests.  Specify `HEROKU_APP` and `HEROKU_API_KEY` on the command line, and then watch your app's logs.
+
+    HEROKU_APP=... HEROKU_API_KEY=... guard
+    heroku logs --app ...
+
+## Authors
+
+Justin Love, [@wondible](http://twitter.com/wondible), [https://github.com/JustinLove](https://github.com/JustinLove)
+
+Ported to Heroku-Api by Fix Peña, [https://github.com/fixr](https://github.com/fixr)
+
+Retry/schedule sets by Matt Anderson [https://github.com/tonkapark](https://github.com/tonkapark) and Thibaud Guillaume-Gentil [https://github.com/jilion](https://github.com/jilion)
+
+## Licence
+
+Released under the [MIT license](http://www.opensource.org/licenses/mit-license.php).

data/examples/complex.rb

@@ -0,0 +1,39 @@
+require 'sidekiq'
+require 'autoscaler/sidekiq'
+require 'autoscaler/heroku_scaler'
+
+heroku = nil
+if ENV['HEROKU_APP']
+  heroku = {}
+  scaleable = %w[default import] - (ENV['ALWAYS'] || '').split(' ')
+  scaleable.each do |queue|
+    heroku[queue] = Autoscaler::HerokuScaler.new(
+      queue,
+      ENV['HEROKU_API_KEY'],
+      ENV['HEROKU_APP'])
+  end
+end
+
+Sidekiq.configure_client do |config|
+  if heroku
+    config.client_middleware do |chain|
+      chain.add Autoscaler::Sidekiq::Client, heroku
+    end
+  end
+end
+
+# define HEROKU_PROCESS in the Procfile:
+#
+#    default: env HEROKU_PROCESS=default bundle exec sidekiq -r ./background/boot.rb
+#    import:  env HEROKU_PROCESS=import bundle exec sidekiq -q import -c 1 -r ./background/boot.rb
+
+Sidekiq.configure_server do |config|
+  config.server_middleware do |chain|
+    if heroku && ENV['HEROKU_PROCESS'] && heroku[ENV['HEROKU_PROCESS']]
+      p "Setting up auto-scaledown"
+      chain.add(Autoscaler::Sidekiq::Server, heroku[ENV['HEROKU_PROCESS']], 60, [ENV['HEROKU_PROCESS']]) # 60 second timeout
+    else
+      p "Not scaleable"
+    end
+  end
+end

data/examples/simple.rb

@@ -0,0 +1,28 @@
+require 'sidekiq'
+require 'autoscaler/sidekiq'
+require 'autoscaler/heroku_scaler'
+
+heroku = nil
+if ENV['HEROKU_APP']
+  heroku = Autoscaler::HerokuScaler.new
+  #heroku.exception_handler = lambda {|exception| MyApp.logger.error(exception)}
+end
+
+Sidekiq.configure_client do |config|
+  if heroku
+    config.client_middleware do |chain|
+      chain.add Autoscaler::Sidekiq::Client, 'default' => heroku
+    end
+  end
+end
+
+Sidekiq.configure_server do |config|
+  config.server_middleware do |chain|
+    if heroku
+      p "Setting up auto-scaledown"
+      chain.add(Autoscaler::Sidekiq::Server, heroku, 60) # 60 second timeout
+    else
+      p "Not scaleable"
+    end
+  end
+end

data/lib/autoscaler.rb

@@ -0,0 +1,5 @@
+require "autoscaler/version"
+
+# Namespace module; no code
+module Autoscaler
+end

data/lib/autoscaler/binary_scaling_strategy.rb

@@ -0,0 +1,26 @@
+module Autoscaler
+  # Strategies determine the target number of workers
+  # The default strategy has a single worker when there is anything, or shuts it down.
+  class BinaryScalingStrategy
+    #@param [integer] active_workers number of workers when in the active state.
+    def initialize(active_workers = 1)
+      @active_workers = active_workers
+    end
+
+    # @param [QueueSystem] system interface to the queuing system
+    # @param [Numeric] event_idle_time number of seconds since a job related event
+    # @return [Integer] target number of workers
+    def call(system, event_idle_time)
+      if active?(system)
+        @active_workers
+      else
+        0
+      end
+    end
+
+    private
+    def active?(system)
+      system.queued > 0 || system.scheduled > 0 || system.retrying > 0 || system.workers > 0
+    end
+  end
+end

data/lib/autoscaler/counter_cache_memory.rb

@@ -0,0 +1,35 @@
+module Autoscaler
+  # Implements a cache for the number of heroku works currently up
+  # Values are stored for short periods in the object
+  class CounterCacheMemory
+    # @param [Numeric] timeout number of seconds to allow before expiration
+    def initialize(timeout = 5)
+      @timeout = timeout
+      @counter = 0
+      @valid_until = Time.now - 1
+    end
+
+    # @param [Numeric] value new counter value
+    def counter=(value)
+      @valid_until = Time.now + @timeout
+      @counter = value
+    end
+
+    # Raised when no block is provided to #counter
+    class Expired < ArgumentError; end
+
+    # Current value.  Uses the Hash#fetch api - pass a block to use in place of expired values or it will raise an exception.
+    def counter
+      return @counter if valid?
+      return yield if block_given?
+      raise Expired
+    end
+
+    private
+    attr_reader :timeout
+
+    def valid?
+      Time.now < @valid_until
+    end
+  end
+end

data/lib/autoscaler/counter_cache_redis.rb

@@ -0,0 +1,50 @@
+module Autoscaler
+  # Implements a cache for the number of heroku works currently up
+  # This permits some web/worker communication, which makes longer timeouts practical.
+  class CounterCacheRedis
+    # @param [Proc, ConnectionPool, Redis client] redis redis interface
+    #   Proc: e.g. Sidekiq.method(:redis)
+    #   ConnectionPool: e.g. what you pass to Sidekiq.redis=
+    #   Redis client: e.g. Redis.connect
+    # @param [Numeric] timeout number of seconds to allow before expiration
+    # @param [String] worker_type the name of the worker type, for cache keys
+    def initialize(redis, timeout = 5 * 60, worker_type = 'worker')
+      @redis = redis
+      @timeout = timeout
+      @worker_type = worker_type
+    end
+
+    # @param [Numeric] value new counter value
+    def counter=(value)
+      redis {|c| c.setex(key, @timeout, value)}
+    end
+
+    # Raised when no block is provided to #counter
+    class Expired < ArgumentError; end
+
+    # Current value.  Uses the Hash#fetch api - pass a block to use in place of expired values or it will raise an exception.
+    def counter
+      value = redis {|c| c.get(key)}
+      return value.to_i if value
+      return yield if block_given?
+      raise Expired
+    end
+
+    private
+    attr_reader :timeout
+
+    def key
+      ['autoscaler', 'workers', @worker_type] * ':'
+    end
+
+    def redis(&block)
+      if @redis.respond_to?(:call)
+        @redis.call(&block)
+      elsif @redis.respond_to?(:with)
+        @redis.with(&block)
+      else
+        block.call(@redis)
+      end
+    end
+  end
+end

data/lib/autoscaler/delayed_shutdown.rb

@@ -0,0 +1,44 @@
+module Autoscaler
+  # This is a sort of middleware that keeps the last worker up for a minimum amount of time
+  class DelayedShutdown
+    # @param [ScalingStrategy] strategy object that makes most decisions
+    # @param [Numeric] timeout number of seconds to stay up after base strategy says zero
+    def initialize(strategy, timeout)
+      @strategy = strategy
+      @timeout = timeout
+      active_now!
+    end
+
+    # @param [QueueSystem] system interface to the queuing system
+    # @param [Numeric] event_idle_time number of seconds since a job related event
+    # @return [Integer] target number of workers
+    def call(system, event_idle_time)
+      target_workers = strategy.call(system, event_idle_time)
+      if target_workers > 0
+        active_now!
+        target_workers
+      elsif time_left?(event_idle_time)
+        1
+      else
+        0
+      end
+    end
+
+    private
+
+    attr_reader :strategy
+    attr_reader :timeout
+
+    def active_now!
+      @activity = Time.now
+    end
+
+    def level_idle_time
+      Time.now - @activity
+    end
+
+    def time_left?(event_idle_time)
+      [event_idle_time, level_idle_time].min < timeout
+    end
+  end
+end

data/lib/autoscaler/heroku_scaler.rb

@@ -0,0 +1,81 @@
+require 'heroku-api'
+require 'autoscaler/counter_cache_memory'
+
+module Autoscaler
+  # Wraps the Heroku API to provide just the interface that we need for scaling.
+  class HerokuScaler
+    # @param [String] type process type this scaler controls
+    # @param [String] key Heroku API key
+    # @param [String] app Heroku app name
+    def initialize(
+        type = 'worker',
+        key = ENV['HEROKU_API_KEY'],
+        app = ENV['HEROKU_APP'])
+      @client = Heroku::API.new(:api_key => key)
+      @type = type
+      @app = app
+      @workers = CounterCacheMemory.new
+    end
+
+    attr_reader :app
+    attr_reader :type
+
+    # Read the current worker count (value may be cached)
+    # @return [Numeric] number of workers
+    def workers
+      @workers.counter {@workers.counter = heroku_get_workers}
+    end
+
+    # Set the number of workers (noop if workers the same)
+    # @param [Numeric] n number of workers
+    def workers=(n)
+      unknown = false
+      current = @workers.counter{unknown = true; 1}
+      if n != current || unknown
+        p "Scaling #{type} to #{n}"
+        heroku_set_workers(n)
+        @workers.counter = n
+      end
+    end
+
+    # Callable object which responds to exceptions during api calls #
+    # @example
+    #   heroku.exception_handler = lambda {|exception| MyApp.logger.error(exception)}
+    #   heroku.exception_handler = lambda {|exception| raise}
+    #   # default
+    #   lambda {|exception|
+    #     p exception
+    #     puts exception.backtrace
+    #   }
+    attr_writer :exception_handler
+
+    # Object which supports #counter and #counter=
+    # Defaults to CounterCacheMemory
+    def counter_cache=(cache)
+      @workers = cache
+    end
+
+    private
+    attr_reader :client
+
+    def heroku_get_workers
+      client.get_ps(app).body.count {|ps| ps['process'].match /#{type}\.\d?/ }
+    rescue Excon::Errors::Error, Heroku::API::Errors::Error => e
+      exception_handler.call(e)
+      0
+    end
+
+    def heroku_set_workers(n)
+      client.post_ps_scale(app, type, n)
+    rescue Excon::Errors::Error, Heroku::API::Errors::Error => e
+      exception_handler.call(e)
+    end
+
+    def exception_handler
+      @exception_handler ||= lambda {|exception|
+        p exception
+        puts exception.backtrace
+      }
+    end
+  end
+end