sidekiq-process_manager 1.0.4 → 1.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cf2d4db525273b507522ba4ec10fb493bb9643bf5311ace3011aa8e1ddb789f9
-  data.tar.gz: 285c66e121b7d64f41b270165596d91c43886866d8c6504c4ca8732d34c378d0
+  metadata.gz: 51d57e09aba69d64225948647f2f95625c668b44eb16e514b5ab4daf8e064e3c
+  data.tar.gz: 73d49c20ce5bf06a8d84a168d8e1afa2b04883ea1656d3d4e0ac96b66855140c
 SHA512:
-  metadata.gz: d10fb3ce00499889150d64baed11dab070c31de8d385ad55ed5665e287034f1ffb04fc4d951ba38f924860d6be58292bc13fdf4fceb74bf06e3011485de3bbfc
-  data.tar.gz: 1b94ddf8bee336e5600e5151fcac282f71648c88a6dabea9024464698ecdfffcdb80e02c22fb865415e3c033e1e38e3901b6e14331b634057b9283775af2ef44
+  metadata.gz: 2f35e8a6ccbbcbaa15fbbbd62658bd79e4337e675a0c4a776e8dc2c6457bb957ed5ba9880336532a68f54e5ea13890917d758d5f84a2395130c0c52a5166a7cc
+  data.tar.gz: dd4fa55ec09584a890e5125d93a64d15924b138583d23e91331a1818bec0c58dfa0cdd81ab58bfab37951a645ed41e149fa04996ec89d11c85c0362204c21be4

data/CHANGELOG.md

@@ -4,25 +4,47 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [1.0.4] - 2021-05-20
+## 1.1.1
+
+### Added
+- Guards to ensure signal processing thread doesn't die.
+
+## 1.1.0
+
+### Added
+- Sidekiq 7 support.
+- Max memory setting to automatically restart processes suffering from memory bloat.
+- Use a notification pipe to handle signals (@KevinCarterDev)
+
+### Removed
+- Sidekiq < 5.0 support.
+- Ruby < 2.5 support.
+
+## 1.0.4
+
 ### Fixed
 - Set $0 to "sidekiq" in preforked process so instrumentation libraries detecting sidekiq server from the command line will work.
 
-## [1.0.3] - 2021-05-19
+## 1.0.3
+
 ### Fixed
-- Restore bin dir to gem distribution
+- Restore bin dir to gem distribution.
+
+## 1.0.2
 
-## [1.0.2] - 2021-05-19
 ### Added
-- Support for sidekiq >= 6.1
+- Support for sidekiq >= 6.1.
 - Set $0 to "sidekiq" so instrumentation libraries detecting sidekiq server from the command line will work.
 
 ### Changed
-- Minimum Ruby version 2.3
+- Minimum Ruby version 2.3.
+
+## 1.0.1
 
-## [1.0.1] - 2020-02-20
 ### Changed
 - Remove auto require of `sidekiq/cli` so `require: false` does not need to be specified in a Gemfile.
 
-## [1.0.0] - 2019-11-27
-- Initial release
+## 1.0.0
+
+### Added
+- Initial release.

data/README.md

@@ -42,6 +42,10 @@ If your application can't be pre-forked, you can at least load the gem files and
 
 For a Rails application, you would normally want to preboot the `config/boot.rb` file.
 
+## Memory Bloat
+
+You can also specify a maximum memory footprint that you want to allow for each child process. You can use this feature to automatically guard against poorly designed workers that bloat the Ruby memory heap. Note that you can also use an external process monitor to kill processes with memory bloat; the process manager will restart any process regardless of how it dies.
+
 ## Usage
 
 Install the gem in your sidekiq process and run it with `bundle exec sidekiq-process-manager` or, if you use [bundle binstubs](https://bundler.io/man/bundle-binstubs.1.html), `bin/sidekiq-process-manager`. Command line arguments are passed through to `sidekiq`. If you want to supply on of the `sidekiq-process_manager` specific options, those options should come first and the `sidekiq` options should appear after a `--` flag
@@ -93,6 +97,18 @@ or
 SIDEKIQ_PREBOOT=config/boot.rb SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
 ```
 
+You can set the maximum memory allowed per sidekiq process with the `--max-memory` argument or with the `SIDEKIQ_MAX_MEMORY` environment variable. You can suffix the value with "m" to specify megabytes or "g" to specify gigabytes.
+
+```bash
+bundle exec sidekiq-process-manager --processes 4 --max-memory 2g
+```
+
+or
+
+```bash
+SIDEKIQ_MAX_MEMORY=2000m SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
+```
+
 ## Alternatives
 
 Any process manager can be an alternative (service, update, systemd, monit, god, etc.).
@@ -103,4 +119,32 @@ The advantages this gem can provide are:
 
 2. Running in the foreground with output going to standard out instead of as daemon can integrate better into containerized environments.
 
-3. Built with sidekiq in mind so signal passing is consistent.
+3. Built with sidekiq in mind so signal passing is consistent with signals used when running a simple sidekiq process.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "sidekiq-process_manager", require: false
+```
+
+Then execute:
+```bash
+$ bundle
+```
+
+Or install it yourself as:
+```bash
+$ gem install sidekiq-process_manager
+```
+
+## Contributing
+
+Open a pull request on [GitHub](https://github.com/bdurand/sidekiq-process_manager).
+
+Please use the [standardrb](https://github.com/testdouble/standard) syntax and lint your code with `standardrb --fix` before submitting.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/VERSION

@@ -0,0 +1 @@
+1.1.1

data/bin/sidekiq-process-manager

@@ -6,10 +6,27 @@ require_relative "../lib/sidekiq-process_manager"
 
 DEFAULT_PROCESS_COUNT = 1
 
+def parse_max_memory(max_memory)
+  value = nil
+
+  matched = max_memory.to_s.match(/\A([\d]+(?:\.[\d]+)?)([mg])\z/i)
+  if matched
+    value = matched[1].to_f
+    if matched[2].downcase == 'm'
+      value *= 1024 * 1024
+    elsif matched[2].downcase == 'g'
+      value *= 1024 * 1024 * 1024
+    end
+  end
+
+  value
+end
+
 options = {
   process_count: Integer(ENV.fetch('SIDEKIQ_PROCESSES', DEFAULT_PROCESS_COUNT)),
   prefork: !ENV.fetch("SIDEKIQ_PREFORK", "").empty?,
   preboot: ENV["SIDEKIQ_PREBOOT"],
+  max_memory: parse_max_memory(ENV["SIDEKIQ_MAX_MEMORY"]),
   mode: nil,
 }
 
@@ -28,6 +45,10 @@ parser = OptionParser.new do |opts|
     options[:preboot] = preboot
   end
 
+  opts.on('--max-memory MEMORY', "Max memory for each process (can also specify with SIDEKIQ_MAX_MEMORY); suffix with m or g to specify megabytes or gigabytes") do |max_memory|
+    options[:max_memory] = parse_max_memory(max_memory)
+  end
+
   opts.on('--testing', "Enable test mode") do |testing|
     options[:mode] = :testing if testing
   end

data/lib/sidekiq/process_manager/manager.rb

@@ -1,13 +1,24 @@
 # frozen_string_literal: true
 
 require "sidekiq"
+require "get_process_mem"
 
 module Sidekiq
   module ProcessManager
+    # Process manager for sidekiq. This class is responsible for starting and monitoring
+    # that the specified number of sidekiq processes are running. It will also forward
+    # signals sent to the main process to the child processes.
     class Manager
       attr_reader :cli
 
-      def initialize(process_count: 1, prefork: false, preboot: nil, mode: nil, silent: false)
+      # Create a new process manager.
+      #
+      # @param process_count [Integer] The number of sidekiq processes to start.
+      # @param prefork [Boolean] If true, the process manager will load the application before forking.
+      # @param preboot [String] If set, the process manager will require the specified file before forking the child processes.
+      # @param mode [Symbol] If set to :testing, the process manager will use a mock CLI.
+      # @param silent [Boolean] If true, the process manager will not output any messages.
+      def initialize(process_count: 1, prefork: false, preboot: nil, max_memory: nil, mode: nil, silent: false)
         require "sidekiq/cli"
 
         # Get the number of processes to fork
@@ -16,19 +27,22 @@ module Sidekiq
 
         @prefork = (prefork && process_count > 1)
         @preboot = preboot if process_count > 1 && !prefork
+        @max_memory = ((max_memory.to_i > 0) ? max_memory.to_i : nil)
 
         if mode == :testing
           require_relative "../../../spec/support/mocks"
           @cli = MockSidekiqCLI.new(silent)
+          @memory_check_interval = 1
         else
           @cli = Sidekiq::CLI.instance
+          @memory_check_interval = 60
         end
 
         @silent = silent
         @pids = []
         @terminated_pids = []
         @started = false
-        @monitor = Monitor.new
+        @mutex = Mutex.new
       end
 
       # Start the process manager. This method will start the specified number
@@ -38,6 +52,8 @@ module Sidekiq
       #
       # Child processes are manged by sending the signals you would normally send
       # to a sidekiq process to the process manager instead.
+      #
+      # @return [void]
       def start
         raise "Process manager already started" if started?
         @started = true
@@ -46,11 +62,26 @@ module Sidekiq
 
         master_pid = ::Process.pid
 
+        @signal_pipe_read, @signal_pipe_write = IO.pipe
+
+        @signal_thread = Thread.new do
+          Thread.current.name = "signal-handler"
+
+          while @signal_pipe_read.wait_readable
+            begin
+              signal = @signal_pipe_read.gets.strip
+              send_signal_to_children(signal.to_sym)
+            rescue => e
+              log_error("Error handling signal #{signal}: #{e.message}")
+            end
+          end
+        end
+
         # Trap signals that will be forwarded to child processes
         [:INT, :TERM, :USR1, :USR2, :TSTP, :TTIN].each do |signal|
           ::Signal.trap(signal) do
             if ::Process.pid == master_pid
-              send_signal_to_children(signal)
+              @signal_pipe_write.puts(signal)
             end
           end
         end
@@ -58,25 +89,33 @@ module Sidekiq
         # Ensure that child processes receive the term signal when the master process exits.
         at_exit do
           if ::Process.pid == master_pid && @process_count > 0
-            @pids.each do |pid|
-              send_signal_to_children(:TERM)
-            end
+            send_signal_to_children(:TERM)
           end
         end
 
+        GC.start
+        GC.compact if GC.respond_to?(:compact)
+        # I'm not sure why, but running GC operations blocks until we try to write some I/O.
+        File.write("/dev/null", "0")
+
         @process_count.times do
           start_child_process!
         end
 
+        start_memory_monitor if @max_memory
+
         log_info("Process manager started with pid #{::Process.pid}")
         monitor_child_processes
         log_info("Process manager #{::Process.pid} exiting")
       end
 
       # Helper to wait on the manager to wait on child processes to start up.
+      #
+      # @param timeout [Integer] The number of seconds to wait for child processes to start.
+      # @return [void]
       def wait(timeout = 5)
-        start_time = Time.now
-        while Time.now < start_time + timeout
+        timeout_time = monotonic_time + timeout
+        while monotonic_time <= timeout_time
           return if @pids.size == @process_count
           sleep(0.01)
         end
@@ -85,16 +124,25 @@ module Sidekiq
       end
 
       # Helper to gracefully stop all child processes.
+      #
+      # @return [void]
       def stop
+        stop_memory_monitor
         @process_count = 0
         send_signal_to_children(:TSTP)
         send_signal_to_children(:TERM)
       end
 
+      # Get all chile process pids.
+      #
+      # @return [Array<Integer>]
       def pids
-        @pids.dup
+        @mutex.synchronize { @pids.dup }
       end
 
+      # Return true if the process manager has started.
+      #
+      # @return [Boolean]
       def started?
         @started
       end
@@ -103,9 +151,9 @@ module Sidekiq
 
       def log_info(message)
         return if @silent
-        if $stdout.tty?
-          $stdout.write("#{message}#{$/}")
-          $stdout.flush
+        if $stderr.tty?
+          $stderr.write("#{message}#{$/}")
+          $stderr.flush
         else
           Sidekiq.logger.info(message)
         end
@@ -121,10 +169,27 @@ module Sidekiq
         end
       end
 
+      def monotonic_time
+        ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+      end
+
+      def sidekiq_options
+        if Sidekiq.respond_to?(:default_configuration)
+          Sidekiq.default_configuration
+        else
+          Sidekiq.options
+        end
+      end
+
       def load_sidekiq
         @cli.parse
-        Sidekiq.options[:daemon] = false
-        Sidekiq.options[:pidfile] = false
+
+        # Disable daemonization and pidfile creation for child processes (sidekiq < 6.0)
+        if Gem::Version.new(Sidekiq::VERSION) < Gem::Version.new("6.0")
+          sidekiq_options[:daemon] = false
+          sidekiq_options[:pidfile] = false
+        end
+
         if @prefork
           log_info("Pre-forking application")
           # Set $0 so instrumentation libraries detecting sidekiq from the command run will work properly.
@@ -148,26 +213,30 @@ module Sidekiq
       end
 
       def set_program_name!
-        $PROGRAM_NAME = "sidekiq process manager #{Sidekiq.options[:tag]} [#{@pids.size} processes]"
+        $PROGRAM_NAME = "sidekiq process manager #{sidekiq_options[:tag]} [#{@pids.size} processes]"
       end
 
       def start_child_process!
-        @pids << fork do
+        pid = fork do
           # Set $0 so instrumentation libraries detecting sidekiq from the command run will work properly.
           $0 = File.join(File.dirname($0), "sidekiq")
           @process_count = 0
           @pids.clear
+          @signal_thread.kill
+          @signal_pipe_read.close
+          @signal_pipe_write.close
           Sidekiq::ProcessManager.run_after_fork_hooks
           @cli.run
         end
-        log_info("Forked sidekiq process with pid #{@pids.last}")
+        @mutex.synchronize { @pids << pid }
+        log_info("Forked sidekiq process with pid #{pid}")
         set_program_name!
       end
 
       def send_signal_to_children(signal)
         log_info("Process manager trapped signal #{signal}")
         @process_count = 0 if signal == :INT || signal == :TERM
-        @pids.each do |pid|
+        pids.each do |pid|
           begin
             log_info("Sending signal #{signal} to sidekiq process #{pid}")
             ::Process.kill(signal, pid)
@@ -175,18 +244,81 @@ module Sidekiq
             log_warning("Error sending signal #{signal} to sidekiq process #{pid}: #{e.inspect}")
           end
         end
+        wait_for_children_to_exit(pids) if @process_count == 0
+      end
+
+      def start_memory_monitor
+        log_info("Starting memory monitor with max memory #{(@max_memory / (1024**2)).round}mb")
+
+        @memory_monitor = Thread.new do
+          Thread.current.name = "memory-monitor"
+          loop do
+            sleep(@memory_check_interval)
+
+            pids.each do |pid|
+              begin
+                memory = GetProcessMem.new(pid)
+                if memory.bytes > @max_memory
+                  log_warning("Kill bloated sidekiq process #{pid}: #{memory.mb.round}mb used")
+                  begin
+                    ::Process.kill(:TERM, pid)
+                  rescue Errno::ESRCH
+                    # The process is already dead
+                  end
+                  break
+                end
+              rescue => e
+                log_warning("Error monitoring memory for sidekiq process #{pid}: #{e.inspect}")
+              end
+            end
+          end
+        end
+      end
+
+      def stop_memory_monitor
+        if defined?(@memory_monitor) && @memory_monitor
+          @memory_monitor.kill
+        end
+      end
+
+      def wait_for_children_to_exit(pids)
+        timeout = monotonic_time + (sidekiq_options[:timeout] || 25).to_f
+        pids.each do |pid|
+          while monotonic_time < timeout
+            break unless process_alive?(pid)
+            sleep(0.01)
+          end
+        end
+
+        pids.each do |pid|
+          begin
+            ::Process.kill(:INT, pid) if process_alive?(pid)
+          rescue
+            # Ignore errors so we can continue to kill other processes.
+          end
+        end
+      end
+
+      def process_alive?(pid)
+        begin
+          ::Process.getpgid(pid)
+          true
+        rescue Errno::ESRCH
+          false
+        end
       end
 
       # Listen for child processes dying and restart if necessary.
       def monitor_child_processes
         loop do
           pid = ::Process.wait
-          @pids.delete(pid)
+          @mutex.synchronize { @pids.delete(pid) }
+
           log_info("Sidekiq process #{pid} exited")
 
           # If there are not enough processes running, start a replacement one.
           if @process_count > @pids.size
-            start_child_process! if @pids.size < @process_count
+            start_child_process!
           end
 
           set_program_name!

data/lib/sidekiq/process_manager/version.rb

@@ -2,6 +2,6 @@
 
 module Sidekiq
   module ProcessManager
-    VERSION = "1.0.4"
+    VERSION = File.read(File.expand_path("../../../VERSION", __dir__)).strip
   end
 end

data/sidekiq-process_manager.gemspec

@@ -1,10 +1,8 @@
 # frozen_string_literal: true
 
-require_relative "lib/sidekiq/process_manager/version"
-
 Gem::Specification.new do |spec|
   spec.name = "sidekiq-process_manager"
-  spec.version = Sidekiq::ProcessManager::VERSION
+  spec.version = File.read(File.expand_path("VERSION", __dir__)).strip
   spec.authors = ["Brian Durand"]
   spec.email = ["bbdurand@gmail.com"]
 
@@ -31,9 +29,10 @@ Gem::Specification.new do |spec|
   spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.required_ruby_version = ">= 2.3"
+  spec.required_ruby_version = ">= 2.5"
 
-  spec.add_dependency "sidekiq", ">= 3.0"
+  spec.add_dependency "sidekiq", ">= 5.0"
+  spec.add_dependency "get_process_mem"
 
   spec.add_development_dependency "bundler"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-process_manager
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 1.1.1
 platform: ruby
 authors:
 - Brian Durand
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-05-20 00:00:00.000000000 Z
+date: 2023-07-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -16,14 +16,28 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '5.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: get_process_mem
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -38,7 +52,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: 
+description:
 email:
 - bbdurand@gmail.com
 executables:
@@ -49,6 +63,7 @@ files:
 - CHANGELOG.md
 - MIT_LICENSE.txt
 - README.md
+- VERSION
 - bin/sidekiq-process-manager
 - lib/sidekiq-process_manager.rb
 - lib/sidekiq/process_manager.rb
@@ -59,7 +74,7 @@ homepage: https://github.com/bdurand/sidekiq-process_manager
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -67,15 +82,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.3'
+      version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
-signing_key: 
+rubygems_version: 3.4.12
+signing_key:
 specification_version: 4
 summary: Process manager for forking and monitoring multiple sidekiq processes.
 test_files: []