resque_starter 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0ab87ab83e696f9a2082b4cb7162dbbdd8a77efd
+  data.tar.gz: ab0e725521c9bf8e8d759224b69867bd7820feaa
+SHA512:
+  metadata.gz: 4ea956678caef390ea95e715362b343052b0c61c95614960ccb6780b7ae6a59600131ddb1b4a7e2039671129ea2f594bbebfbaad3aeb3309306a79d5add38a99
+  data.tar.gz: 6b9894349095bc5774f9b1feab545f495aba696a668faa12d9f6be6387c67eb7c44db49b3e2caceae427160d361c26a161c78036c7dfeb598b63d170b25c4bbc

data/.gitignore

@@ -0,0 +1,16 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log
+/tmp/log/*
+/tmp/pids/*

data/Gemfile

@@ -0,0 +1,5 @@
+source 'https://rubygems.org'
+
+gemspec
+gem 'pry'
+gem 'pry-nav'

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2016 sonots
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,119 @@
+# ResqueStarter
+
+[Resque](https://github.com/resque/resque) is widely used Redis-backed Ruby library for creating background jobs. [ResqueStarter](https://github.com/sonots/resque_starter) is a tool to start and manage multiple resque workers supporing graceful shutdown and restart, leveraging the [Copy-on-write (COW)](https://en.wikipedia.org/wiki/Copy-on-write).
+
+```
+PID   COMMAND
+14814 resque_starter # preload app
+14815  \_ resque:work[1]
+14816  \_ resque:work[2]
+14817  \_ resque:work[3]
+```
+
+# Background
+
+[Resque](https://github.com/resque/resque) provides [resque:workers](https://github.com/resque/resque#running-multiple-workers) task to run multiple resque workers, but it is only for development purpose as [code comments](https://github.com/resque/resque/blob/c295da9de0034b20ce79600e9f54fb279695f522/lib/resque/tasks.rb#L23-L38) says.
+It also provides an example configuration of [god](http://godrb.com/) as [resque.god](https://github.com/resque/resque/blob/c295da9de0034b20ce79600e9f54fb279695f522/examples/god/resque.god), but it does not allow us to leverage Copy-on-write (CoW) to save memory of preloaded application.
+
+# Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'resque_starter'
+```
+
+And then execute:
+
+```
+$ bundle
+```
+
+Or install it yourself as:
+
+```
+$ gem install resque_starter
+```
+
+# Configuration
+
+Please see [example/resque.conf.rb](./example/resque.conf.rb)
+
+You can configure logger, pid file, number of concurrency, dequeue interval, queue lists.
+
+# Usage
+
+```
+bundle exec resque_starter -c /path/to/resque.conf.rb
+```
+
+# Signals
+
+Resque starter responds to a few different signals:
+
+* TERM / INT - Quick shutdown, kills all workers immediately then exit
+* QUIT - Graceful shutdown, waits for workers to finish processing then exit
+* USR1 - Send USR1 to all workers, which immediately kill worker's child but don't exit
+* USR2 - Send USR2 to all workers, which don't start to process any new jobs
+* CONT - Send CONT to all workers, which start to process new jobs again after a USR2
+* TTIN - Increment the number of worker processes by one
+* TTOU - Decrement the number of worker processes by one with QUIT
+
+# Graceful restart
+
+Resque starter itself does not support graceful restart, yet. But, graceful restart can be done with [server-starter](https://github.com/sonots/ruby-server-starter).
+
+Example configuration is available at [server-starter/example/resque](https://github.com/sonots/ruby-server-starter/blob/master/example/resque). See `start_server` and `config/resque.conf.rb` files.
+
+**HOW IT WORKS**
+
+On receiving HUP signal, server starter creates a new `resque_starter` (master) process.
+The new `resque_starter` (master) process forks a new resque:work.
+On `after_fork`, send `TTOU` to old `resque_starter` (master) process to gracefully shutdown one old resque:work.
+By repeating this procedure, new `resque_starter` process can be gracefully restarted.
+The number of working resque workers will be suppressed up to `concurrency + 1` in this way.
+
+**ILLUSTRATION**
+
+On bootup:
+
+```
+PID   COMMAND
+14813 server_starter
+14814  \_ resque_starter
+14815      \_ resque:work[1]
+14816      \_ resque:work[2]
+14817      \_ resque:work[3]
+```
+
+Send HUP:
+
+```
+PID   COMMAND
+14813 server_starter
+14814  \_ resque_starter (old)
+14815      \_ resque:work[1] (dies)
+14816      \_ resque:work[2]
+14817      \_ resque:work[3]
+14818  \_ resque_starter (new)
+14819      \_ resque:work[1]
+```
+
+Finally:
+
+```
+PID   COMMAND
+14813 server_starter
+14818  \_ resque_starter (new)
+14819      \_ resque:work[1]
+14820      \_ resque:work[2]
+14821      \_ resque:work[3]
+```
+
+# Contributing
+
+1. Fork it ( https://github.com/sonots/resque_starter/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/bin/resque_starter

@@ -0,0 +1 @@
+resque_starter.rb

data/bin/resque_starter.rb

@@ -0,0 +1,32 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require_relative '../lib/resque_starter'
+
+opts = {
+}
+OptionParser.new do |opt|
+  opt.on(
+    '--config=file',
+    'path to config file',
+  ) {|v| opts[:config_file] = v }
+  opt.on(
+    '--version',
+    'print version',
+  ) {|v| puts ResqueStarter::VERSION; exit 0 }
+  
+  opt.parse!(ARGV)
+
+  define_method(:usage) do |msg = nil|
+    puts opt.to_s
+    puts "error: #{msg}" if msg
+    exit 1
+  end
+end
+
+if opts[:config_file].nil?
+  usage 'Option --config is required'
+end
+
+starter = ResqueStarter.new(opts[:config_file])
+starter.start_resque

data/bin/start_resque

@@ -0,0 +1 @@
+resque_starter.rb

data/bin/start_resque.rb

@@ -0,0 +1 @@
+resque_starter.rb

data/example/resque.conf.rb

@@ -0,0 +1,43 @@
+# Number of concurrency (resque workers)
+concurrency 2
+
+# By default, resque_starter logs to stdout
+# logger ::Logger.new(::File.expand_path('../../tmp/log/resque_starter.log', __FILE__))
+# logger.level = ::Logger::INFO
+# logger.formatter = ResqueStarter::Logger::Formatter.new
+
+# Stores pid for resque starter (master) itself
+pid_file ::File.expand_path('../../tmp/pids/resque_starter.pid', __FILE__)
+
+# Status file stores pids of resque workers, etc
+status_file ::File.expand_path('../../tmp/pids/resque_starter.stat', __FILE__)
+
+# Watching queues of resque workers in priority orders
+# Same with QUEUES environment variables for rake resque:work
+# See https://github.com/resque/resque#priorities-and-queue-lists
+#
+# Default obtains from @queue class instance variables of worker class
+# See https://github.com/resque/resque#overview
+queues ['high', 'low']
+
+# Polling frequency (default: 5)
+# Same with INTERVAL environment variables for rake resque:work
+# See https://github.com/resque/resque#polling-frequency
+dequeue_interval 0.1
+
+# Preload rails application codes before forking to save memory by CoW
+# require ::File.expand_path('../../config/environment', __FILE__)
+# Rails.application.eager_load!
+
+before_fork do |starter, worker, worker_nr|
+  # the following is highly recomended for Rails + "preload_app true"
+  # as there's no need for the master process to hold a connection
+  defined?(ActiveRecord::Base) and
+    ActiveRecord::Base.connection.disconnect!
+end
+
+after_fork do |starter, worker, worker_nr|
+  # the following is *required* for Rails + "preload_app true",
+  defined?(ActiveRecord::Base) and
+    ActiveRecord::Base.establish_connection
+end

data/lib/resque_starter/config.rb

@@ -0,0 +1,60 @@
+require 'resque_starter/version'
+
+class ResqueStarter::Config
+  def initialize(config_file)
+    @config_file = config_file
+    # TIPS: resque worker ENV configuration
+    # self.verbose = ENV['LOGGING'] || ENV['VERBOSE']
+    # self.very_verbose = ENV['VVERBOSE']
+    # self.term_timeout = ENV['RESQUE_TERM_TIMEOUT'] || 4.0
+    # self.term_child = ENV['TERM_CHILD']
+    # self.graceful_term = ENV['GRACEFUL_TERM']
+    # self.run_at_exit_hooks = ENV['RUN_AT_EXIT_HOOKS']
+    reload
+  end
+
+  def reload
+    @set = Hash.new
+    @set[:concurrency] = 1
+    @set[:dequeue_interval] = 5.0
+    @set[:logger] = ResqueStarter::Logger.new(STDOUT)
+    instance_eval(File.read(@config_file), @config_file)
+  end
+
+  %i[
+    concurrency
+    logger
+    pid_file
+    status_file
+    queues
+    dequeue_interval
+  ].each do |name|
+    define_method(name) do |val = nil|
+      if val
+        @set[name] = val
+      else
+        @set[name]
+      end
+    end
+  end
+
+  def [](key)
+    @set[key]
+  end
+
+  def before_fork(&block)
+    if block_given?
+      @set[:before_fork] = block
+    else
+      @set[:before_fork]
+    end
+  end
+
+  def after_fork(&block)
+    if block_given?
+      @set[:after_fork] = block
+    else
+      @set[:after_fork]
+    end
+  end
+end

data/lib/resque_starter/logger.rb

@@ -0,0 +1,33 @@
+require 'resque_starter/version'
+require 'logger'
+
+class ResqueStarter::Logger < ::Logger
+  class Formatter
+    def call(severity, time, progname, msg)
+      "#{time.iso8601} [#{severity}] #{format_message(msg)}\n"
+    end
+
+    def format_message(message)
+      case message
+      when ::Exception
+        e = message
+        "#{e.class} (#{e.message})\\n  #{e.backtrace.join("\\n  ")}"
+      else
+        message.to_s
+      end
+    end
+  end
+
+  def initialize(logdev, shift_age = 0, shift_size = 1048576)
+    logdev = STDOUT if logdev == 'STDOUT'
+    logdev = STDERR if logdev == 'STDERR'
+    super(logdev, shift_age, shift_size)
+    @formatter = Formatter.new
+    self.level = 'INFO'
+  end
+
+  def level=(level)
+    level = eval("::Logger::#{level.upcase}") if level.is_a?(String)
+    super(level)
+  end
+end

data/lib/resque_starter/version.rb

@@ -0,0 +1,3 @@
+class ResqueStarter
+  VERSION = "0.1.0"
+end

data/lib/resque_starter.rb

@@ -0,0 +1,180 @@
+require 'yaml'
+require 'resque'
+require 'resque_starter/version'
+require 'resque_starter/config'
+require 'resque_starter/logger'
+
+class ResqueStarter
+  attr_reader :config, :logger, :old_workers, :num_workers
+
+  def initialize(config_file)
+    @config = ResqueStarter::Config.new(config_file)
+
+    @old_workers = {}
+    @num_workers = config[:concurrency]
+    @self_read, @self_write = IO.pipe
+
+    # open pid file
+    if config[:pid_file]
+      begin
+        File.open(config[:pid_file], "w") do |fh|
+          fh.puts $$
+        end
+      rescue => e
+        @logger.error "failed to open file:#{config[:pid_file]}"
+        exit(1)
+      end
+      at_exit { File.unlink config[:pid_file] rescue nil }
+    end
+
+    # logger
+    @logger = @config[:logger]
+
+    # create guard that removes the status file
+    if config[:status_file]
+      at_exit { File.unlink config[:status_file] rescue nil }
+    end
+  end
+
+  def start_resque
+    @logger.info "starting resque starter (master) #{Process.pid}"
+    maintain_worker_count
+
+    install_signal_handler
+    @handle_thr = start_handle_thread
+    while true
+      break unless @handle_thr.alive? # dies if @shutdown and @old_workers.empty?
+      begin
+        pid, status = Process.waitpid2(-1)
+        @logger.info "resque worker #{pid} died, status:#{status.exitstatus}"
+        @self_write.puts(pid)
+      rescue Errno::ECHILD
+        @handle_thr.kill if @shutdown # @old_workers should be empty
+        sleep 0.1 # avoid busy loop for no child by TTOU
+      end
+    end
+
+    @logger.info "shutting down resque starter (master) #{Process.pid}"
+  end
+
+  def install_signal_handler
+    %w(TERM INT QUIT USR1 USR2 CONT TTIN TTOU).each do |sig|
+      trap(sig) { @self_write.puts(sig) }
+    end
+  end
+
+  def start_handle_thread
+    Thread.new {
+      while readable_io = IO.select([@self_read])
+        s = readable_io.first[0].gets.strip
+        if pid = (Integer(s) rescue nil)
+          handle_waitpid2(pid)
+        else
+          handle_signal(s)
+        end
+      end
+    }
+  end
+
+  def update_status_file
+    return unless @config[:status_file]
+    # pid => worker_nr
+    status = {'old_workers' => @old_workers}.to_yaml
+    File.write(@config[:status_file], status)
+  end
+
+  def handle_waitpid2(pid)
+    @old_workers.delete(pid)
+    update_status_file
+    if @shutdown
+      @handle_thr.kill if @old_workers.empty?
+    else
+      maintain_worker_count
+    end
+  end
+
+  # Resque starter (master process) responds to a few different signals:
+  # * TERM / INT - Quick shutdown, kills all workers immediately then exit
+  # * QUIT - Graceful shutdown, waits for workers to finish processing then exit
+  # * USR1 - Send USR1 to all workers, which immediately kill worker's child but don't exit
+  # * USR2 - Send USR2 to all workers, which don't start to process any new jobs
+  # * CONT - Send CONT to all workers, which start to process new jobs again after a USR2
+  # * TTIN - Increment the number of worker processes by one
+  # * TTOU - Decrement the number of worker processes by one with QUIT
+  def handle_signal(sig)
+    pids = @old_workers.keys.sort
+    msg = "received #{sig}, "
+
+    # Resque workers respond to a few different signals:
+    # * TERM / INT - Immediately kill child then exit
+    # * QUIT - Wait for child to finish processing then exit
+    # * USR1 - Immediately kill child but don't exit
+    # * USR2 - Don't start to process any new jobs
+    # * CONT - Start to process new jobs again after a USR2
+    case sig
+    when 'TERM', 'INT' 
+      @logger.info msg << "immediately kill all resque workers then exit:#{pids}"
+      @shutdown = true
+      Process.kill(sig, *pids)
+    when 'QUIT'
+      @logger.info msg << "wait for all resque workers to finish processing then exit:#{pids}"
+      @shutdown = true
+      Process.kill(sig, *pids)
+    when 'USR1'
+      @logger.info msg << "immediately kill the child of all resque workers:#{pids}"
+      Process.kill(sig, *pids)
+    when 'USR2'
+      @logger.info msg << "don't start to process any new jobs:#{pids}"
+      Process.kill(sig, *pids)
+    when 'CONT'
+      @logger.info msg << "start to process new jobs again after USR2:#{pids}"
+      Process.kill(sig, *pids)
+    when 'TTIN'
+      @num_workers += 1
+      @logger.info msg << "increment the number of resque workers:#{@num_workers}"
+      maintain_worker_count # ToDo: forking from a thread would be unsafe
+    when 'TTOU'
+      @num_workers -= 1 if @num_workers > 0
+      @logger.info msg << "decrement the number of resque workers:#{@num_workers}"
+      maintain_worker_count
+    end
+  rescue Errno::ESRCH
+  rescue => e
+    @logger.error(e)
+    exit(1)
+  end
+
+  def maintain_worker_count
+    return if (off = @old_workers.size - @num_workers) == 0
+    return spawn_missing_workers if off < 0
+    @old_workers.dup.each_pair {|pid, nr|
+      if nr >= @num_workers
+        Process.kill(:QUIT, pid) rescue nil
+      end
+    }
+  end
+
+  def spawn_missing_workers
+    worker_nr = -1
+    until (worker_nr += 1) == @num_workers
+      next if @old_workers.value?(worker_nr)
+      worker = Resque::Worker.new(*(config[:queues]))
+      config[:before_fork].call(self, worker, worker_nr)
+      GC.start
+      if pid = fork
+        @old_workers[pid] = worker_nr
+        update_status_file
+        @logger.info "starting new resque worker #{pid}"
+      else # child
+        @self_read.close rescue nil
+        @self_write.close rescue nil
+        config[:after_fork].call(self, worker, worker_nr)
+        worker.work(config[:dequeue_interval])
+        exit
+      end
+    end
+  rescue => e
+    @logger.error(e)
+    exit(1)
+  end
+end

data/resque_starter.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'resque_starter/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "resque_starter"
+  spec.version       = ResqueStarter::VERSION
+  spec.authors       = ["Naotoshi Seo"]
+  spec.email         = ["sonots@gmail.com"]
+  spec.summary       = %q{Start and manage multiple resque workers}
+  spec.description   = %q{Start and manage multiple resque workers.}
+  spec.homepage      = "https://github.com/sonots/resque_starter"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "resque"
+
+  spec.add_development_dependency "bundler", "~> 1.7"
+  spec.add_development_dependency "rake", "~> 10.0"
+end

metadata

@@ -0,0 +1,107 @@
+--- !ruby/object:Gem::Specification
+name: resque_starter
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Naotoshi Seo
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-08-20 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: resque
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+description: Start and manage multiple resque workers.
+email:
+- sonots@gmail.com
+executables:
+- resque_starter
+- resque_starter.rb
+- start_resque
+- start_resque.rb
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/resque_starter
+- bin/resque_starter.rb
+- bin/start_resque
+- bin/start_resque.rb
+- example/resque.conf.rb
+- lib/resque_starter.rb
+- lib/resque_starter/config.rb
+- lib/resque_starter/logger.rb
+- lib/resque_starter/version.rb
+- resque_starter.gemspec
+- tmp/log/.gitkeep
+- tmp/pids/.gitkeep
+homepage: https://github.com/sonots/resque_starter
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Start and manage multiple resque workers
+test_files: []