sidekiq-process_manager 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e3d1b1462578e76a03c0908d2e04fe30162a06947e5176e478b7cbf51f797c2d
+  data.tar.gz: bd2f58e84676cac41f9a44bbb4299038b30912e319d793164eccac1a79dff2c7
+SHA512:
+  metadata.gz: e0dd9349a7ec9d5c6ca15728218ef087d824e5b074ac1725393948a00834765d1291a7b3a775b39a6b9cb4905892a2878d19adefb405183db916c9622f882a17
+  data.tar.gz: adf75b664e86f909341822c65954995def3729205c0d73e69a766d69541d60bce84241330b7d9b7917eb778e50400ebb44a67ac9a4e7fa766a9b61b56f46ffa6

data/MIT_LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2019 Brian Durand
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,105 @@
+# Sidekiq::ProcessManager
+
+[![Build Status](https://travis-ci.com/bdurand/sidekiq-process_manager.svg?branch=master)](https://travis-ci.com/bdurand/sidekiq-process_manager)
+[![Maintainability](https://api.codeclimate.com/v1/badges/ed89164d480af0e1442e/maintainability)](https://codeclimate.com/github/bdurand/sidekiq-process_manager/maintainability)
+
+This gem provides a command line script for managing [sidekiq](https://github.com/mperham/sidekiq) processes. It starts up a process that then forks multiple sidekiq processes and manages their life cycle. This is important for large sidekiq installations, since without it on MRI ruby, sidekiq will only use one CPU core. By starting multiple processes you make all cores available.
+
+The sidekiq processes can all be managed by sending signals to the manager process. This process simply forwards the signals on to the child processes, allowing you to control the sidekiq processes as you normally would.
+
+If one of the sidekiq processes dies unexpectedly, the process manager automatically starts a new sidekiq process to replace it.
+
+## Pre-Forking
+
+You can use pre-forking to improve memory utilization on the child sidekiq processes. This is similar to using pre-forking in a web server like puma or unicorn. You application will be pre-loaded by the master process and the child sidekiq processes will utilize the loaded code via copy-on-write memory. The overall effect is that you should be able to run more sidekiq processes in a lower memory footprint.
+
+One issue with pre-forking is that any file descriptors (including network connections) your application has open when it forks will be shared between all the processes. This can lead to race conditions and other problems. To fix it, you must close and reopen all database connections, etc. after the child sidekiq processes have been forked.
+
+To do this, your application will need to add an initializer with calls to `Sidekiq::ProcessManager.after_fork` and `Sidekiq::ProcessManager.before_fork`.
+
+The `before_fork` hook is called on the master process right before it starts forking processes. You can use this to close connections on the master process that you don't need.
+
+The `after_fork` hook is called after a forked sidekiq process has initialized the application. You can use this to re-establish connections so that each process gets it's own streams.
+
+At a minimum, you'll probably want the following hooks to close and re-open the ActiveRecord database connection:
+
+```ruby
+Sidekiq::ProcessManager.before_fork do
+  ActiveRecord::Base.connection.disconnect!
+end
+
+Sidekiq::ProcessManager.after_fork do
+  ActiveRecord::Base.connection.reconnect!
+end
+```
+
+If you're already using a pre-forking web server, you'll need to do most of the same things for sidekiq as well.
+
+## Pre-Booting
+
+If your application can't be pre-forked, you can at load the gem files and libraries your application depends on instead by specifying a preboot file. This file will be loaded by the master process and any code loaded will be copy-on-write shared with the child processes.
+
+For a Rails application, you would normally want to preboot the `config/boot.rb` file.
+
+## Usage
+
+Install the gem in your sidekiq process and run it with `bundle exec sidekiq-process-manager` or, if you use [bundle binstubs](https://bundler.io/man/bundle-binstubs.1.html), `bin/sidekiq-process-manager`. Command line arguments are passed through to `sidekiq`. If you want to supply on of the `sidekiq-process_manager` specific options, those options should come first and the `sidekiq` options should appear after a `--` flag
+
+```bash
+bundle exec sidekiq-process-manager -C config/sidekiq.yml
+```
+
+or
+
+
+```bash
+bundle exec sidekiq-process-manager --no-prefork -- -C config/sidekiq.yml
+```
+
+You can specify the number of sidekiq processes to run with the `--processes` argument or with the `SIDEKIQ_PROCESSES` environment variable. The default number of processes is 1.
+
+```bash
+bundle exec sidekiq-process-manager --processes 4
+```
+
+or
+
+```bash
+SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
+```
+
+You can turn pre-forking on or off with the --prefork or --no-prefork flag. You can also specify to turn on pre-forking with the `SIDEKIQ_PREFORK` environment variable.
+
+```bash
+bundle exec sidekiq-process-manager --processes 4 --prefork
+```
+
+or
+
+```bash
+SIDEKIQ_PREFORK=1 SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
+```
+
+You can turn pre-booting on with the `--preboot` argument or with the `SIDEKIQ_PREBOOT` environment variable.
+
+```bash
+bundle exec sidekiq-process-manager --processes 4 --preboot config/boot.rb
+```
+
+or
+
+```bash
+SIDEKIQ_PREBOOT=config/boot.rb SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
+```
+
+## Alternatives
+
+Any process manager can be an alternative (service, update, systemd, monit, god, etc.).
+
+The advantages this gem can provide are:
+
+1. Pre-forking can be useful on systems with many cores but not enough memory to run enough sidekiq processes to use them all.
+
+2. Running in the foreground with output going to standard out instead of as daemon can integrate better into containerized environments.
+
+3. Built with sidekiq in mind so signal passing is consistent.

data/bin/sidekiq-process-manager

@@ -0,0 +1,67 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'optparse'
+require_relative "../lib/sidekiq-process_manager"
+
+DEFAULT_PROCESS_COUNT = 1
+
+options = {
+  process_count: Integer(ENV.fetch('SIDEKIQ_PROCESSES', DEFAULT_PROCESS_COUNT)),
+  prefork: !ENV.fetch("SIDEKIQ_PREFORK", "").empty?,
+  preboot: ENV["SIDEKIQ_PREBOOT"],
+  mode: nil,
+}
+
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: sidekiq-process-manager [options] [--] [sidekiq options]"
+
+  opts.on('--processes PROCESSES', Integer, "Number of processes to spin up (can also specify with SIDEKIQ_PROCESSES)") do |count|
+    options[:process_count] = count
+  end
+
+  opts.on('--[no-]prefork', "Use prefork for spinning up sidekiq processes (can also specify with SIDEKIQ_PREFORK)") do |prefork|
+    options[:prefork] = prefork
+  end
+
+  opts.on('--preboot FILE', "Preboot the processes (can also specify with SIDEKIQ_PREBOOT)") do |preboot|
+    options[:preboot] = preboot
+  end
+
+  opts.on('--testing', "Enable test mode") do |testing|
+    options[:mode] = :testing if testing
+  end
+
+  opts.on("--help", "Prints this help") do
+    puts opts
+    exit
+  end
+
+  opts.separator(<<~DESCR)
+
+  After the manager options, pass in any options for the sidekiq processes.
+  Additionally, passing in the optional `--` will explicitly end the manager options and begin the sidekiq opts.
+  E.g.
+      $ sidekiq-process-manager --no-prefork -- -C config/sidekiq.rb
+      Calls sidekiq with `sidekiq -C config/sidekiq.rb`
+  DESCR
+end
+
+sidekiq_args = []
+begin
+  parser.order!(ARGV) { |nonopt| sidekiq_args << nonopt }
+rescue OptionParser::InvalidOption => err
+  # Handle the case where a user doesn't put in the `--` to separate the args
+  sidekiq_args.concat(err.args)
+end
+
+ARGV[0, 0] = sidekiq_args
+
+begin
+  manager = Sidekiq::ProcessManager::Manager.new(**options)
+  manager.start
+rescue => e
+  STDERR.puts e.message
+  STDERR.puts e.backtrace.join($/)
+  exit 1
+end

data/lib/sidekiq/process_manager/manager.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+require "sidekiq"
+require "sidekiq/cli"
+
+module Sidekiq
+  module ProcessManager
+    class Manager
+
+      attr_reader :cli
+
+      def initialize(process_count: 1, prefork: false, preboot: nil, mode: nil, silent: false)
+        # Get the number of processes to fork
+        @process_count = process_count
+        raise ArgumentError.new("Process count must be greater than 1") if @process_count < 1
+
+        @prefork = (prefork && process_count > 1)
+        @preboot = preboot if process_count > 1 && !prefork
+
+        if mode == :testing
+          require_relative "../../../spec/support/mocks"
+          @cli = MockSidekiqCLI.new(silent)
+        else
+          @cli = Sidekiq::CLI.instance
+        end
+
+        @silent = silent
+        @pids = []
+        @terminated_pids = []
+        @started = false
+        @monitor = Monitor.new
+      end
+
+      # Start the process manager. This method will start the specified number
+      # of sidekiq processes and monitor them. It will only exit once all child
+      # processes have exited. If a child process dies unexpectedly, it will be
+      # restarted.
+      #
+      # Child processes are manged by sending the signals you would normally send
+      # to a sidekiq process to the process manager instead.
+      def start
+        raise "Process manager already started" if started?
+        @started = true
+
+        load_sidekiq
+
+        master_pid = ::Process.pid
+
+        # Trap signals that will be forwarded to child processes
+        [:INT, :TERM, :USR1, :USR2, :TSTP, :TTIN].each do |signal|
+          ::Signal.trap(signal) do
+            if ::Process.pid == master_pid
+              send_signal_to_children(signal)
+            end
+          end
+        end
+
+        # Ensure that child processes receive the term signal when the master process exits.
+        at_exit do
+          if ::Process.pid == master_pid && @process_count > 0
+            @pids.each do |pid|
+              send_signal_to_children(:TERM)
+            end
+          end
+        end
+
+        @process_count.times do
+          start_child_process!
+        end
+
+        log_info("Process manager started with pid #{::Process.pid}")
+        monitor_child_processes
+        log_info("Process manager #{::Process.pid} exiting")
+      end
+
+      # Helper to wait on the manager to wait on child processes to start up.
+      def wait(timeout = 5)
+        start_time = Time.now
+        while Time.now < start_time + timeout
+          return if @pids.size == @process_count
+          sleep(0.01)
+        end
+
+        raise Timeout::Error.new("child processes failed to start in #{timeout} seconds")
+      end
+
+      # Helper to gracefully stop all child processes.
+      def stop
+        @process_count = 0
+        send_signal_to_children(:TSTP)
+        send_signal_to_children(:TERM)
+      end
+
+      def pids
+        @pids.dup
+      end
+
+      def started?
+        @started
+      end
+
+      private
+
+      def log_info(message)
+        return if @silent
+        if $stdout.tty?
+          $stdout.write("#{message}#{$/}")
+          $stdout.flush
+        else
+          Sidekiq.logger.info(message)
+        end
+      end
+
+      def log_warning(message)
+        return if @silent
+        if $stderr.tty?
+          $stderr.write("#{message}#{$/}")
+          $stderr.flush
+        else
+          Sidekiq.logger.warn(message)
+        end
+      end
+
+      def load_sidekiq
+        @cli.parse
+        Sidekiq.options[:daemon] = false
+        Sidekiq.options[:pidfile] = false
+        if @prefork
+          log_info("Pre-forking application")
+          @cli.send(:boot_system)
+          Sidekiq::ProcessManager.run_before_fork_hooks
+        elsif @preboot && !@preboot.empty?
+          if ::File.exist?(@preboot)
+            require ::File.expand_path(@preboot).sub(/\.rb\Z/, "")
+          else
+            log_warning("Could not find preboot file #{@preboot}")
+          end
+        end
+      end
+
+      def set_program_name!
+        $PROGRAM_NAME = "sidekiq process manager #{Sidekiq.options[:tag]} [#{@pids.size} processes]"
+      end
+
+      def start_child_process!
+        @pids << fork do
+          @process_count = 0
+          @pids.clear
+          Sidekiq::ProcessManager.run_after_fork_hooks
+          @cli.run
+        end
+        log_info("Forked sidekiq process with pid #{@pids.last}")
+        set_program_name!
+      end
+
+      def send_signal_to_children(signal)
+        log_info("Process manager trapped signal #{signal}")
+        @process_count = 0 if (signal == :INT || signal == :TERM)
+        @pids.each do |pid|
+          begin
+            log_info("Sending signal #{signal} to sidekiq process #{pid}")
+            ::Process.kill(signal, pid)
+          rescue => e
+            log_warning("Error sending signal #{signal} to sidekiq process #{pid}: #{e.inspect}")
+          end
+        end
+      end
+
+      # Listen for child processes dying and restart if necessary.
+      def monitor_child_processes
+        loop do
+          pid = ::Process.wait
+          @pids.delete(pid)
+          log_info("Sidekiq process #{pid} exited")
+
+          # If there are not enough processes running, start a replacement one.
+          if @process_count > @pids.size
+            start_child_process! if @pids.size < @process_count
+          end
+
+          set_program_name!
+
+          if @pids.empty?
+            break
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sidekiq/process_manager/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module ProcessManager
+    VERSION = "1.0.0"
+  end
+end

data/lib/sidekiq/process_manager.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+require_relative "process_manager/version"
+require_relative "process_manager/manager"
+
+module Sidekiq
+  module ProcessManager
+
+    class << self
+      def before_fork(&block)
+        @before_fork ||= []
+        @before_fork << block
+      end
+
+      def after_fork(&block)
+        @after_fork ||= []
+        @after_fork << block
+      end
+
+      def run_before_fork_hooks
+        if defined?(@before_fork) && @before_fork
+          @before_fork.each do |block|
+            block.call
+          end
+        end
+        @before_fork = nil
+      end
+
+      def run_after_fork_hooks
+        if defined?(@after_fork) && @after_fork
+          @after_fork.each do |block|
+            block.call
+          end
+        end
+        @after_fork = nil
+      end
+    end
+
+  end
+end

data/lib/sidekiq-process_manager.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative "sidekiq/process_manager"

data/sidekiq-process_manager.gemspec

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require_relative "lib/sidekiq/process_manager/version"
+
+Gem::Specification.new do |spec|
+  spec.name          = "sidekiq-process_manager"
+  spec.version       = Sidekiq::ProcessManager::VERSION
+  spec.authors       = ["Brian Durand"]
+  spec.email         = ["bbdurand@gmail.com"]
+
+  spec.summary       = "Process manager for forking and monitoring multiple sidekiq processes."
+  spec.homepage      = "https://github.com/bdurand/sidekiq-process_manager"
+  spec.license       = "MIT"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  ignore_files = %w(
+    .gitignore
+    .travis.yml
+    Appraisals
+    Gemfile
+    Gemfile.lock
+    Rakefile
+    gemfiles/
+    spec/
+  )
+  spec.files         = Dir.chdir(File.expand_path('..', __FILE__)) do
+    `git ls-files -z`.split("\x0").reject{ |f| ignore_files.any?{ |path| f.start_with?(path) } }
+  end
+  spec.bindir        = "bin"
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.required_ruby_version = '>= 2.2.2'
+
+  spec.add_dependency "sidekiq", ">= 3.0"
+
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency "appraisal"
+end

metadata

@@ -0,0 +1,122 @@
+--- !ruby/object:Gem::Specification
+name: sidekiq-process_manager
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Brian Durand
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2019-11-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: appraisal
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- bbdurand@gmail.com
+executables:
+- sidekiq-process-manager
+extensions: []
+extra_rdoc_files: []
+files:
+- MIT_LICENSE.txt
+- README.md
+- bin/sidekiq-process-manager
+- lib/sidekiq-process_manager.rb
+- lib/sidekiq/process_manager.rb
+- lib/sidekiq/process_manager/manager.rb
+- lib/sidekiq/process_manager/version.rb
+- sidekiq-process_manager.gemspec
+homepage: https://github.com/bdurand/sidekiq-process_manager
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.2.2
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.3
+signing_key: 
+specification_version: 4
+summary: Process manager for forking and monitoring multiple sidekiq processes.
+test_files: []