sidekiq-heroku-autoscale 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3e8ad2ab9e6bdd803cecdf5f0148c5ef5cc25549fca06387e89ed449ff5ec635
+  data.tar.gz: 8997e2e8461d6c08735aa51b7bc9099ee17ef7aa4a42fda33ad96d9105fa12dd
+SHA512:
+  metadata.gz: b1c57075a0d77844e06fe1d7940cdaa4243c74d9e9fbed2f1012583b9cf6dd41d72ac07c4bca032b0f814714315528450a18cca6f6332a09cb17a0c70cdc3b1f
+  data.tar.gz: d1911ced4e2fc584c334f920aec14b93778e0bbbc3871458c7020da9a55857a919fbe8817240437b5b58011f792de9d1a7158d38d4d8807c70bc5a6353c136a2

data/README.md

@@ -0,0 +1,165 @@
+# Sidekiq Heroku Autoscale plugin
+
+This [Sidekiq](https://github.com/mperham/sidekiq) plugin allows Heroku dynos to be started, stopped, and scaled based on job workload. Why? Because running non-stop Sidekiq dynos on Heroku can rack up unnecessary costs for apps with modest background processing needs.
+
+This is a self-acknowledged rewrite of the [autoscaler](https://github.com/JustinLove/autoscaler) project. While this plugin borrows many foundation concepts from _autoscaler_, it rewrites core operations to address several logistical concerns and enable reporting through a web UI.
+
+![Web UI](web-preview.png)
+
+Tested with Sidekiq 6, but should be compatible with other recent Sidekiq versions.
+
+## How it works
+
+This plugin operates by tapping into Sidekiq startup hooks and middleware.
+
+- Whenever a server is started or a job is queued, the appropriate process manager is called on to adjust its scale. Adjustments are throttled across process instances (dynos) so that the Heroku API is only called once every N seconds – 10 by default.
+
+- When workload demands more dynos, scale will adjust directly upward to target capacity.
+
+- As workload diminishes, scale will adjust downward one dyno at a time. When downscaling a process, the highest numbered dyno (ex: `worker.1`, `worker.2`, etc...) will be quieted and then removed from the formation. This combines Heroku's [autoscaling logic](https://devcenter.heroku.com/articles/scaling#autoscaling-logic) with Sidekiq's [quieting strategy](https://github.com/mperham/sidekiq/wiki/Signals#tstp).
+
+## Gem installation
+
+```ruby
+gem 'sidekiq'
+gem 'sidekiq-heroku-autoscale'
+```
+
+If you're not using Rails, you'll need to require `sidekiq-heroku-autoscale` after `sidekiq`.
+
+## Environment config
+
+You'll need to generate a Heroku platform API token that enables your app to adjust its own dyno formation. This can be done through the Heroku CLI with:
+
+```shell
+heroku authorizations:create
+```
+
+Copy the `Token` value and add it along with your app's name as environment variables in your app:
+
+```shell
+SIDEKIQ_HEROKU_AUTOSCALE_API_TOKEN=<token>
+SIDEKIQ_HEROKU_AUTOSCALE_APP=<app-name>
+```
+
+The Heroku Autoscaler plugin will automatically check for these two environment variables. You'll also find some setup suggestions in Sidekiq's [Heroku deployment](https://github.com/mperham/sidekiq/wiki/Deployment#heroku) docs. Specifically, you'll want to include the `-t 25` option in your Procfile's Sidekiq command to maximize process quietdown time:
+
+```shell
+web: bundle exec rails start
+worker: bundle exec sidekiq -t 25
+```
+
+## Plugin config
+
+Add a configuration file for the Heroku Autoscale plugin. YAML works well. A simple configuration with one `worker` process (named in your Procfile) that monitors all Sidekiq queues and starts/stops in the presence of work looks like this:
+
+**config/sidekiq_heroku_autoscale.yml**
+
+```yaml
+app_name: test-app
+processes:
+  worker:
+    system:
+      watch_queues: *
+      include_retrying: true
+      include_scheduled: false
+    scale:
+      mode: binary
+      max_dynos: 1
+```
+
+Then, add an initializer to hand your configuration off to the plugin:
+
+**config/initializers/sidekiq.rb**
+
+```ruby
+config = YAML.load_file('<path/to/config.yml>')
+Sidekiq::HerokuAutoscale.init(config)
+```
+
+A more advanced configuration with multiple process types that watch specific queues would look like this – where `first` and `second` are two Heroku process types:
+
+```yaml
+api_token: <optional - for dynamic insertion only!>
+app_name: test-app
+throttle: 20
+history: 7200
+processes:
+  first:
+    system:
+      watch_queues:
+        - default
+        - low
+      include_retrying: false
+      include_scheduled: false
+    scale:
+      mode: binary
+      max_dynos: 2
+    quiet_buffer: 15
+
+  second:
+    system:
+      watch_queues:
+        - high
+      include_retrying: false
+      include_scheduled: false
+    scale:
+      mode: linear
+      max_dynos: 5
+      workers_per_dyno: 20
+      min_factor: 1
+```
+
+**Config Options**
+
+- `api_token:` optional, same as `SIDEKIQ_HEROKU_AUTOSCALE_API_TOKEN`. Always prefer the ENV variable, or dynamically insert this.
+- `app_name:` optional, same as `SIDEKIQ_HEROKU_AUTOSCALE_APP`.
+- `throttle:` number of seconds to throttle between scale adjustments. The default is 10, so the Heroku API will only be hit once every ten seconds regardless of how many time the process is pinged during that timeframe. This value also dictates the tick frequency on the web UI history graph.
+- `history:` number of seconds to track history in the web UI. The default is 3600, or 1 hour. The history graph renders ticks using the history duration divided by throttle time – so 3600 seconds of history on a 10 second throttle produce 360 data points. Therefore, it's best to keep these settings in modest proportions to one another. You'll probably be sad if you try to display days or weeks of history.
+- `processes:` a list of Heroku process types named in your Procfile. For example, `worker` or `sidekiq`.
+- `process.system.watch_queues:` a list of Sidekiq queues to watch for work, or `*` for all queues. Queue names must be mutually exclusive to avoid collisions. That means a queue name may only appear once across all processes, and that `*` (all) may not be combined with other queue names.
+- `process.system.include_retrying:` specifies if the Sidekiq retry set should be included while assessing workload. Watching retries may cause undesirable levels of uptime.
+- `process.system.include_scheduled:` specifies if the Sidekiq scheduled set should be included while assessing workload. Watching scheduled jobs may cause undesirable levels of idle uptime. Also, no new jobs will be scheduled unless Sidekiq is running.
+- `process.scale.mode:` accepts "binary" (on/off) or "linear" (scaled to workload).
+- `process.scale.max_dynos:` maximum allowed concurrent dynos. In binary mode, this will be the fixed operating capacity. In linear mode, this will be the maximum extent that dynos may scale up to.
+- `process.scale.workers_per_dyno:` Linear mode only. This specifies the anticipated workforce per dyno to calculate scale around. This should generally align with Sidekiq's `concurrency` setting.
+- `process.quiet_buffer:` number of seconds to quiet a dyno (stopping it from taking on new work) before downscaling its process. This buffer occurs _before_ reducing the number of dynos for a given process type. After downscale, you may configure an [additional quietdown threshold](https://github.com/mperham/sidekiq/wiki/Deployment#heroku). Note: during this quiet buffer your formation includes a decomissioned dyno, which is awkward – thus no other scale adjustments (up or down) are allowed until the quieted dyno has been dropped. Be accordingly judicious with this buffer.
+
+## Web UI
+
+The web UI is an optional extension of Sidekiq's web UI. To activate it, just require `sidekiq/heroku_autoscale/web` after the base `sidekiq/web`, and then mount `Sidekiq::Web` as normal:
+
+```ruby
+require 'sidekiq/web'
+require 'sidekiq/heroku_autoscale/web'
+
+Rails.application.routes.draw do
+  mount Sidekiq::Web, at: '/sidekiq'
+end
+```
+
+## Tests
+
+Nothing fancy...
+
+```bash
+# start a redis server
+redis-server test/redis_test.conf
+
+# then run tests in another terminal window
+bundle exec rake test
+```
+
+### Contributors
+
+- Justin Love [@wondible](http://twitter.com/wondible), [https://github.com/JustinLove](https://github.com/JustinLove)
+- Benjamin Kudria [https://github.com/bkudria](https://github.com/bkudria)
+- claudiofullscreen [https://github.com/claudiofullscreen](https://github.com/claudiofullscreen)
+- Fix Peña [https://github.com/fixr](https://github.com/fixr)
+- Gabriel Givigier Guimarães [https://github.com/givigier](https://github.com/givigier)
+- Matt Anderson [https://github.com/tonkapark](https://github.com/tonkapark)
+- Thibaud Guillaume-Gentil [https://github.com/jilion](https://github.com/jilion)
+
+## Licence
+
+Sidekiq Heroku Autoscale plugin is released under the [MIT license](https://opensource.org/licenses/MIT).

data/lib/sidekiq-heroku-autoscale.rb

@@ -0,0 +1,2 @@
+require 'sidekiq'
+require 'sidekiq/heroku_autoscale'

data/lib/sidekiq/heroku_autoscale.rb

@@ -0,0 +1,69 @@
+require 'sidekiq/heroku_autoscale/heroku_app'
+require 'sidekiq/heroku_autoscale/middleware'
+require 'sidekiq/heroku_autoscale/poll_interval'
+require 'sidekiq/heroku_autoscale/process'
+require 'sidekiq/heroku_autoscale/queue_system'
+require 'sidekiq/heroku_autoscale/scale_strategy'
+require 'sidekiq/heroku_autoscale/version'
+
+module Sidekiq
+  module HerokuAutoscale
+
+    class << self
+      def app
+        @app
+      end
+
+      def init(options)
+        options = options.transform_keys(&:to_sym)
+        @app = HerokuApp.new(options)
+
+        ::Sidekiq.logger.warn('Heroku platform API is not configured for Sidekiq::HerokuAutoscale') unless @app.live?
+
+        # configure sidekiq queue server
+        ::Sidekiq.configure_server do |config|
+          config.on(:startup) do
+            dyno_name = ENV['DYNO']
+            next unless dyno_name
+
+            process = @app.process_by_name(dyno_name.split('.').first)
+            next unless process
+
+            process.ping!
+          end
+
+          config.server_middleware do |chain|
+            chain.add(Middleware, @app)
+          end
+
+          # for jobs that queue other jobs...
+          config.client_middleware do |chain|
+            chain.add(Middleware, @app)
+          end
+        end
+
+        # configure sidekiq app client
+        ::Sidekiq.configure_client do |config|
+          config.client_middleware do |chain|
+            chain.add(Middleware, @app)
+          end
+        end
+
+        # immedaitely wake all processes during client launch
+        @app.ping! unless ::Sidekiq.server?
+
+        @app
+      end
+
+      def exception_handler
+        @exception_handler ||= lambda { |ex|
+          p ex
+          puts ex.backtrace
+        }
+      end
+
+      attr_writer :exception_handler
+    end
+
+  end
+end

data/lib/sidekiq/heroku_autoscale/heroku_app.rb

@@ -0,0 +1,144 @@
+require 'platform-api'
+
+module Sidekiq
+  module HerokuAutoscale
+
+    class HerokuApp
+      attr_reader :app_name, :throttle, :history
+
+      # Builds process managers based on configuration (presumably loaded from YAML)
+      def initialize(config)
+        config = JSON.parse(JSON.generate(config), symbolize_names: true)
+
+        api_token = config[:api_token] || ENV['SIDEKIQ_HEROKU_AUTOSCALE_API_TOKEN']
+        @app_name = config[:app_name] || ENV['SIDEKIQ_HEROKU_AUTOSCALE_APP']
+        @throttle = config[:throttle] || 10
+        @history = config[:history] || 60 * 60 # 1 hour
+        @client = api_token ? PlatformAPI.connect_oauth(api_token) : nil
+
+        @processes_by_name = {}
+        @processes_by_queue = {}
+
+        config[:processes].each_pair do |name, opts|
+          process = Process.new(
+            app_name: @app_name,
+            name: name,
+            client: @client,
+            throttle: @throttle,
+            history: @history,
+            **opts.slice(:system, :scale, :quiet_buffer)
+          )
+          @processes_by_name[name.to_s] = process
+
+          process.queue_system.watch_queues.each do |queue_name|
+            # a queue may only be managed by a single heroku process type (to avoid scaling conflicts)
+            # thus, raise an error over duplicate queue names or when "*" isn't exclusive
+            if @processes_by_queue.key?(queue_name) || @processes_by_queue.key?('*') || (queue_name == '*' && @processes_by_queue.keys.any?)
+              raise ArgumentError, 'watched queues must be exclusive to a single Heroku process type'
+            end
+            @processes_by_queue[queue_name] = process
+          end
+        end
+      end
+
+      # checks if there's a live Heroku client setup
+      def live?
+        !!@client
+      end
+
+      # pings all processes in the application
+      # useful for requesting live updates
+      def ping!
+        processes.each(&:ping!)
+      end
+
+      def processes
+        @processes ||= @processes_by_name.values
+      end
+
+      def process_names
+        @process_names ||= @processes_by_name.keys
+      end
+
+      def queue_names
+        @queue_names ||= @processes_by_queue.keys
+      end
+
+      def process_by_name(process_name)
+        @processes_by_name[process_name]
+      end
+
+      def process_for_queue(queue_name)
+        @processes_by_queue[queue_name] || @processes_by_queue['*']
+      end
+
+      def stats
+        histories = history_stats
+        processes.each_with_object({}) do |process, memo|
+          memo[process.name] = {
+            dynos: process.dynos,
+            status: process.status,
+            updated: process.updated_at.to_s,
+            history: histories[process.name],
+          }
+        end
+      end
+
+      def history_stats(now=Time.now.utc)
+        # calculate a series time to anchor graph ticks on
+        # the series snaps to thresholds of N (throttle duration)
+        series_time = (now.to_f / @throttle).floor * @throttle
+        num_ticks = (@history / @throttle).floor
+        first_tick = series_time - @throttle * num_ticks
+
+        # all ticks is a hash of timestamp keys to plot
+        all_ticks = Array.new(num_ticks)
+          .each_with_index.map { |v, i| (first_tick + @throttle * i).to_s }
+          .each_with_object({}) { |tick, memo| memo[tick] = nil }
+
+        # get current and previous history collections for each process
+        # history pages snap to thresholds of M (history duration)
+        current_page = (now.to_f / @history).floor * @history
+        previous_page = current_page - @history
+        history_pages = ::Sidekiq.redis do |c|
+          c.pipelined do
+            processes.each do |process|
+              c.hgetall("#{ process.cache_key }:#{ previous_page }")
+              c.hgetall("#{ process.cache_key }:#{ current_page }")
+            end
+          end
+        end
+
+        history_by_process = {}
+        history_pages.each_slice(2).each_with_index do |(a, b), i|
+          process = processes[i]
+
+          # flatten all history pages into a single collection
+          ticks = all_ticks
+            .merge(a.merge!(b))
+            .map { |k, v| [k.to_i, v ? v.to_i : nil] }
+            .sort_by { |tick| tick[0] }
+
+          # separate the older stats from the current history timeframe
+          past_ticks, present_ticks = ticks.partition { |tick| tick[0] < first_tick }
+
+          # select a running value starting point
+          # run from the end of past history, or beginning of present history, or current dynos
+          value = past_ticks.last || present_ticks.detect { |tick| !!tick[1] }
+          value = value ? value[1] : process.dynos
+
+          # assign a running value across all ticks
+          present_ticks.each do |tick|
+            tick[1] ||= value
+            value = tick[1]
+          end
+
+          history_by_process[process.name] = present_ticks
+        end
+
+        history_by_process
+      end
+    end
+
+  end
+end

data/lib/sidekiq/heroku_autoscale/middleware.rb

@@ -0,0 +1,21 @@
+module Sidekiq
+  module HerokuAutoscale
+
+    class Middleware
+      def initialize(app)
+        @app = app
+      end
+
+      def call(worker_class, item, queue, _=nil)
+        result = yield
+
+        if process = @app.process_for_queue(queue)
+          process.ping!
+        end
+
+        result
+      end
+    end
+
+  end
+end

data/lib/sidekiq/heroku_autoscale/poll_interval.rb

@@ -0,0 +1,34 @@
+module Sidekiq
+  module HerokuAutoscale
+
+    class PollInterval
+      def initialize(method_name, before_update: 0, after_update: 0)
+        @method_name = method_name
+        @before_update = before_update
+        @after_update = after_update
+        @requests = {}
+      end
+
+      def call(process)
+        return unless process
+        @requests[process.name] ||= process
+        poll!
+      end
+
+      def poll!
+        @thread ||= Thread.new do
+          begin
+            while @requests.size > 0
+              sleep(@before_update) if @before_update > 0
+              @requests.reject! { |n, p| p.send(@method_name) }
+              sleep(@after_update) if @after_update > 0
+            end
+          ensure
+            @thread = nil
+          end
+        end
+      end
+    end
+
+  end
+end