sidekiq-process_manager 1.1.1 → 1.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 51d57e09aba69d64225948647f2f95625c668b44eb16e514b5ab4daf8e064e3c
4
- data.tar.gz: 73d49c20ce5bf06a8d84a168d8e1afa2b04883ea1656d3d4e0ac96b66855140c
3
+ metadata.gz: 24bc40ee7ef3f0ade56a7c116d509cc7cbb88c4f15c1781a6c065be88ebe4a0c
4
+ data.tar.gz: 9c3704f538d9cb990aa4cb55e4fc2ce8f4b59c60aa0de92d0a95f1cca44e30fc
5
5
  SHA512:
6
- metadata.gz: 2f35e8a6ccbbcbaa15fbbbd62658bd79e4337e675a0c4a776e8dc2c6457bb957ed5ba9880336532a68f54e5ea13890917d758d5f84a2395130c0c52a5166a7cc
7
- data.tar.gz: dd4fa55ec09584a890e5125d93a64d15924b138583d23e91331a1818bec0c58dfa0cdd81ab58bfab37951a645ed41e149fa04996ec89d11c85c0362204c21be4
6
+ metadata.gz: fe31823918abde6d3128ada00c22aa60e25e3d16c7a3ebe85fbeaa25d20aa228ad5aee0ed68c0746876ae98bebbc951ea46725c0a339d583566befa82a2d1b1c
7
+ data.tar.gz: 39f00a80aaef0fc0e936f592a8b592a497ba697c366787d006924e1df73239db424a74d354bada8fadebb6f51e8bea87c44b4807b5b56a994a29902a2930526c
data/CHANGELOG.md CHANGED
@@ -4,6 +4,29 @@ All notable changes to this project will be documented in this file.
4
4
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
5
5
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
6
 
7
+ ## 1.1.3
8
+
9
+ ### Added
10
+ - The interval between memory checks can now be set with the `--memory-check-interval` command line argument or the `SIDEKIQ_MEMORY_CHECK_INTERVAL` environment variable. It still defaults to 60 seconds.
11
+
12
+ ### Fixed
13
+ - Child processes now reset the signal handlers inherited from the process manager to their default behavior when they are forked. The inherited handlers do nothing outside of the master process, so any signal sent to a child process before sidekiq installed its own handlers was silently discarded and the process would only be stopped by the SIGKILL failsafe.
14
+ - Fixed race condition where a child process forked to replace a dead process while a TERM or INT signal was being handled would never receive the shutdown signal, leaving the process manager hung waiting on it.
15
+ - Child processes are now reaped while waiting for them to exit at shutdown, with a timeout, so the process manager can no longer hang forever waiting on zombie processes if it exits abnormally. All processes are polled on each pass so that one slow process cannot consume the entire timeout, and any that are still running when the timeout expires are killed with SIGKILL and reaped rather than being orphaned.
16
+ - Child processes spawned by libraries or application code (e.g. `ps` invoked by the get_process_mem gem) are no longer mistaken for sidekiq processes by the process monitor.
17
+ - The SIGKILL failsafe now uses the manager's list of running processes to determine liveness instead of a system call, removing the possibility of signaling an unrelated process if the operating system reuses a process id.
18
+ - The signal handling thread exits cleanly instead of busy-looping if the signal pipe is closed.
19
+ - Fixed the validation error message when the process count is less than one.
20
+ - The `sidekiq-process-manager` command line now accepts a max memory value in plain bytes and aborts with an error message on invalid values instead of silently ignoring them.
21
+
22
+ ## 1.1.2
23
+
24
+ ### Added
25
+ - Added thread to monitor child processes to make sure they exit after a SIGTERM signal has been sent. If a process does not exit after the configured Sidekiq timeout time, then it will be killed with a SIGKILL signal.
26
+
27
+ ### Changed
28
+ - A SIGINT sent to the manager process will sent SIGTERM to the child processes to give them a chance to shutdown gracefully.
29
+
7
30
  ## 1.1.1
8
31
 
9
32
  ### Added
data/README.md CHANGED
@@ -1,14 +1,14 @@
1
1
  # Sidekiq::ProcessManager
2
2
 
3
3
  [![Continuous Integration](https://github.com/bdurand/sidekiq-process_manager/actions/workflows/continuous_integration.yml/badge.svg)](https://github.com/bdurand/sidekiq-process_manager/actions/workflows/continuous_integration.yml)
4
- [![Maintainability](https://api.codeclimate.com/v1/badges/ed89164d480af0e1442e/maintainability)](https://codeclimate.com/github/bdurand/sidekiq-process_manager/maintainability)
5
4
  [![Ruby Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://github.com/testdouble/standard)
5
+ [![Gem Version](https://badge.fury.io/rb/sidekiq-process_manager.svg)](https://badge.fury.io/rb/sidekiq-process_manager)
6
6
 
7
7
  This gem provides a command line script for managing [sidekiq](https://github.com/mperham/sidekiq) processes. It starts up a process that then forks multiple sidekiq processes and manages their life cycle. This is important for large sidekiq installations, since without it on MRI ruby, sidekiq will only use one CPU core. By starting multiple processes you make all cores available.
8
8
 
9
9
  The sidekiq processes can all be managed by sending signals to the manager process. This process simply forwards the signals on to the child processes, allowing you to control the sidekiq processes as you normally would.
10
10
 
11
- If one of the sidekiq processes dies unexpectedly, the process manager automatically starts a new sidekiq process to replace it.
11
+ If one of the sidekiq processes exits unexpectedly, the process manager automatically starts a new sidekiq process to replace it.
12
12
 
13
13
  ## Pre-Forking
14
14
 
@@ -44,7 +44,7 @@ For a Rails application, you would normally want to preboot the `config/boot.rb`
44
44
 
45
45
  ## Memory Bloat
46
46
 
47
- You can also specify a maximum memory footprint that you want to allow for each child process. You can use this feature to automatically guard against poorly designed workers that bloat the Ruby memory heap. Note that you can also use an external process monitor to kill processes with memory bloat; the process manager will restart any process regardless of how it dies.
47
+ You can also specify a maximum memory footprint that you want to allow for each child process. You can use this feature to automatically guard against poorly designed workers that bloat the Ruby memory heap. Note that you can also use an external process monitor to kill processes with memory bloat; the process manager will restart any process regardless of how it exits.
48
48
 
49
49
  ## Usage
50
50
 
@@ -109,6 +109,18 @@ or
109
109
  SIDEKIQ_MAX_MEMORY=2000m SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
110
110
  ```
111
111
 
112
+ Memory is checked once a minute by default. You can change how often with the `--memory-check-interval` argument or with the `SIDEKIQ_MEMORY_CHECK_INTERVAL` environment variable. The value is in seconds. Note that only one process is restarted per check, so the interval also controls how quickly multiple bloated processes are recycled.
113
+
114
+ ```bash
115
+ bundle exec sidekiq-process-manager --processes 4 --max-memory 2g --memory-check-interval 10
116
+ ```
117
+
118
+ or
119
+
120
+ ```bash
121
+ SIDEKIQ_MEMORY_CHECK_INTERVAL=10 SIDEKIQ_MAX_MEMORY=2g SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
122
+ ```
123
+
112
124
  ## Alternatives
113
125
 
114
126
  Any process manager can be an alternative (service, update, systemd, monit, god, etc.).
@@ -139,6 +151,20 @@ Or install it yourself as:
139
151
  $ gem install sidekiq-process_manager
140
152
  ```
141
153
 
154
+ ## Test Application
155
+
156
+ There is a test application in the `test_app` directory that can be used to test the process manager. You can run it with:
157
+
158
+ ```bash
159
+ $ cd test_app
160
+ $ bundle install
161
+ $ bin/run
162
+ ```
163
+
164
+ This will bring up a valkey docker instance, sidekiq cluster, and rack application that shows the status of the sidekiq processes.
165
+
166
+ Then visit [http://localhost:9292](http://localhost:9292) to see the status of the sidekiq processes.
167
+
142
168
  ## Contributing
143
169
 
144
170
  Open a pull request on [GitHub](https://github.com/bdurand/sidekiq-process_manager).
data/VERSION CHANGED
@@ -1 +1 @@
1
- 1.1.1
1
+ 1.1.3
@@ -7,19 +7,23 @@ require_relative "../lib/sidekiq-process_manager"
7
7
  DEFAULT_PROCESS_COUNT = 1
8
8
 
9
9
  def parse_max_memory(max_memory)
10
- value = nil
11
-
12
- matched = max_memory.to_s.match(/\A([\d]+(?:\.[\d]+)?)([mg])\z/i)
13
- if matched
14
- value = matched[1].to_f
15
- if matched[2].downcase == 'm'
16
- value *= 1024 * 1024
17
- elsif matched[2].downcase == 'g'
18
- value *= 1024 * 1024 * 1024
19
- end
10
+ Sidekiq::ProcessManager.parse_max_memory(max_memory)
11
+ rescue ArgumentError => e
12
+ abort(e.message)
13
+ end
14
+
15
+ def parse_memory_check_interval(interval)
16
+ interval = interval.to_s.strip
17
+ return nil if interval.empty?
18
+
19
+ begin
20
+ interval = Float(interval)
21
+ rescue ArgumentError, TypeError
22
+ abort("Invalid memory check interval value: #{interval.inspect}")
20
23
  end
24
+ abort("Memory check interval must be greater than 0") unless interval > 0
21
25
 
22
- value
26
+ interval
23
27
  end
24
28
 
25
29
  options = {
@@ -27,6 +31,7 @@ options = {
27
31
  prefork: !ENV.fetch("SIDEKIQ_PREFORK", "").empty?,
28
32
  preboot: ENV["SIDEKIQ_PREBOOT"],
29
33
  max_memory: parse_max_memory(ENV["SIDEKIQ_MAX_MEMORY"]),
34
+ memory_check_interval: parse_memory_check_interval(ENV["SIDEKIQ_MEMORY_CHECK_INTERVAL"]),
30
35
  mode: nil,
31
36
  }
32
37
 
@@ -45,10 +50,14 @@ parser = OptionParser.new do |opts|
45
50
  options[:preboot] = preboot
46
51
  end
47
52
 
48
- opts.on('--max-memory MEMORY', "Max memory for each process (can also specify with SIDEKIQ_MAX_MEMORY); suffix with m or g to specify megabytes or gigabytes") do |max_memory|
53
+ opts.on('--max-memory MEMORY', "Max memory for each process (can also specify with SIDEKIQ_MAX_MEMORY); value is in bytes, or suffix with m or g to specify megabytes or gigabytes") do |max_memory|
49
54
  options[:max_memory] = parse_max_memory(max_memory)
50
55
  end
51
56
 
57
+ opts.on('--memory-check-interval SECONDS', "Seconds between memory checks when a max memory is set (can also specify with SIDEKIQ_MEMORY_CHECK_INTERVAL); defaults to #{Sidekiq::ProcessManager::Manager::DEFAULT_MEMORY_CHECK_INTERVAL} seconds") do |interval|
58
+ options[:memory_check_interval] = parse_memory_check_interval(interval)
59
+ end
60
+
52
61
  opts.on('--testing', "Enable test mode") do |testing|
53
62
  options[:mode] = :testing if testing
54
63
  end
@@ -2,6 +2,7 @@
2
2
 
3
3
  require "sidekiq"
4
4
  require "get_process_mem"
5
+ require "timeout"
5
6
 
6
7
  module Sidekiq
7
8
  module ProcessManager
@@ -9,6 +10,12 @@ module Sidekiq
9
10
  # that the specified number of sidekiq processes are running. It will also forward
10
11
  # signals sent to the main process to the child processes.
11
12
  class Manager
13
+ # Signals that are trapped by the process manager and forwarded to the child processes.
14
+ SIGNALS = [:INT, :TERM, :USR1, :USR2, :TSTP, :TTIN].freeze
15
+
16
+ # Number of seconds between memory checks when a max memory is set.
17
+ DEFAULT_MEMORY_CHECK_INTERVAL = 60
18
+
12
19
  attr_reader :cli
13
20
 
14
21
  # Create a new process manager.
@@ -16,31 +23,38 @@ module Sidekiq
16
23
  # @param process_count [Integer] The number of sidekiq processes to start.
17
24
  # @param prefork [Boolean] If true, the process manager will load the application before forking.
18
25
  # @param preboot [String] If set, the process manager will require the specified file before forking the child processes.
26
+ # @param memory_check_interval [Numeric] The number of seconds between memory checks
27
+ # when a max memory is set.
19
28
  # @param mode [Symbol] If set to :testing, the process manager will use a mock CLI.
20
29
  # @param silent [Boolean] If true, the process manager will not output any messages.
21
- def initialize(process_count: 1, prefork: false, preboot: nil, max_memory: nil, mode: nil, silent: false)
30
+ def initialize(process_count: 1, prefork: false, preboot: nil, max_memory: nil, memory_check_interval: nil, mode: nil, silent: false)
22
31
  require "sidekiq/cli"
23
32
 
24
33
  # Get the number of processes to fork
25
34
  @process_count = process_count
26
- raise ArgumentError.new("Process count must be greater than 1") if @process_count < 1
35
+ raise ArgumentError.new("Process count must be greater than 0") if @process_count < 1
27
36
 
28
- @prefork = (prefork && process_count > 1)
37
+ @prefork = prefork && process_count > 1
29
38
  @preboot = preboot if process_count > 1 && !prefork
30
39
  @max_memory = ((max_memory.to_i > 0) ? max_memory.to_i : nil)
31
40
 
41
+ if !memory_check_interval.nil? && memory_check_interval.to_f <= 0
42
+ raise ArgumentError.new("Memory check interval must be greater than 0")
43
+ end
44
+
32
45
  if mode == :testing
33
46
  require_relative "../../../spec/support/mocks"
34
47
  @cli = MockSidekiqCLI.new(silent)
35
- @memory_check_interval = 1
48
+ @memory_check_interval = (memory_check_interval || 1).to_f
49
+ @exit_grace_period = 0.5
36
50
  else
37
51
  @cli = Sidekiq::CLI.instance
38
- @memory_check_interval = 60
52
+ @memory_check_interval = (memory_check_interval || DEFAULT_MEMORY_CHECK_INTERVAL).to_f
53
+ @exit_grace_period = 5
39
54
  end
40
55
 
41
56
  @silent = silent
42
57
  @pids = []
43
- @terminated_pids = []
44
58
  @started = false
45
59
  @mutex = Mutex.new
46
60
  end
@@ -69,18 +83,22 @@ module Sidekiq
69
83
 
70
84
  while @signal_pipe_read.wait_readable
71
85
  begin
72
- signal = @signal_pipe_read.gets.strip
86
+ signal = @signal_pipe_read.gets
87
+ break if signal.nil?
88
+
89
+ signal = signal.strip
73
90
  send_signal_to_children(signal.to_sym)
74
91
  rescue => e
75
- log_error("Error handling signal #{signal}: #{e.message}")
92
+ log_warning("Error sending signal #{signal} to child processes: #{e.message}")
76
93
  end
77
94
  end
78
95
  end
79
96
 
80
97
  # Trap signals that will be forwarded to child processes
81
- [:INT, :TERM, :USR1, :USR2, :TSTP, :TTIN].each do |signal|
98
+ SIGNALS.each do |signal|
82
99
  ::Signal.trap(signal) do
83
100
  if ::Process.pid == master_pid
101
+ signal = :TERM if signal == :INT
84
102
  @signal_pipe_write.puts(signal)
85
103
  end
86
104
  end
@@ -88,15 +106,19 @@ module Sidekiq
88
106
 
89
107
  # Ensure that child processes receive the term signal when the master process exits.
90
108
  at_exit do
91
- if ::Process.pid == master_pid && @process_count > 0
92
- send_signal_to_children(:TERM)
109
+ if ::Process.pid == master_pid
110
+ if @process_count > 0
111
+ send_signal_to_children(:TERM)
112
+ end
113
+ wait_for_children_to_exit
114
+ log_info("Process manager exiting")
93
115
  end
94
116
  end
95
117
 
96
118
  GC.start
97
119
  GC.compact if GC.respond_to?(:compact)
98
120
  # I'm not sure why, but running GC operations blocks until we try to write some I/O.
99
- File.write("/dev/null", "0")
121
+ File.write(File::NULL, "0")
100
122
 
101
123
  @process_count.times do
102
124
  start_child_process!
@@ -104,9 +126,8 @@ module Sidekiq
104
126
 
105
127
  start_memory_monitor if @max_memory
106
128
 
107
- log_info("Process manager started with pid #{::Process.pid}")
129
+ log_info("Process manager started")
108
130
  monitor_child_processes
109
- log_info("Process manager #{::Process.pid} exiting")
110
131
  end
111
132
 
112
133
  # Helper to wait on the manager to wait on child processes to start up.
@@ -116,7 +137,7 @@ module Sidekiq
116
137
  def wait(timeout = 5)
117
138
  timeout_time = monotonic_time + timeout
118
139
  while monotonic_time <= timeout_time
119
- return if @pids.size == @process_count
140
+ return if pids.size == @process_count
120
141
  sleep(0.01)
121
142
  end
122
143
 
@@ -213,11 +234,20 @@ module Sidekiq
213
234
  end
214
235
 
215
236
  def set_program_name!
216
- $PROGRAM_NAME = "sidekiq process manager #{sidekiq_options[:tag]} [#{@pids.size} processes]"
237
+ $PROGRAM_NAME = "sidekiq process manager #{sidekiq_options[:tag]} [#{pids.size} processes]"
217
238
  end
218
239
 
219
240
  def start_child_process!
220
241
  pid = fork do
242
+ # Child processes inherit the signal handlers installed by the process
243
+ # manager. Those handlers only do anything in the master process, so
244
+ # leaving them in place would silently discard any signal sent to the
245
+ # child before sidekiq installs its own handlers. Restore the default
246
+ # behavior until then so signals are never swallowed.
247
+ SIGNALS.each do |signal|
248
+ ::Signal.trap(signal, "DEFAULT")
249
+ end
250
+
221
251
  # Set $0 so instrumentation libraries detecting sidekiq from the command run will work properly.
222
252
  $0 = File.join(File.dirname($0), "sidekiq")
223
253
  @process_count = 0
@@ -228,27 +258,47 @@ module Sidekiq
228
258
  Sidekiq::ProcessManager.run_after_fork_hooks
229
259
  @cli.run
230
260
  end
231
- @mutex.synchronize { @pids << pid }
261
+ shutting_down = false
262
+ @mutex.synchronize do
263
+ @pids << pid
264
+ shutting_down = @process_count < 1
265
+ end
232
266
  log_info("Forked sidekiq process with pid #{pid}")
267
+ # A shutdown signal may have been forwarded to the other processes while
268
+ # this one was being forked; it needs to get the signal as well.
269
+ send_signal_to_pid(:TERM, pid) if shutting_down
233
270
  set_program_name!
234
271
  end
235
272
 
236
273
  def send_signal_to_children(signal)
237
274
  log_info("Process manager trapped signal #{signal}")
238
- @process_count = 0 if signal == :INT || signal == :TERM
239
- pids.each do |pid|
240
- begin
241
- log_info("Sending signal #{signal} to sidekiq process #{pid}")
242
- ::Process.kill(signal, pid)
243
- rescue => e
244
- log_warning("Error sending signal #{signal} to sidekiq process #{pid}: #{e.inspect}")
275
+ child_pids = @mutex.synchronize do
276
+ @process_count = 0 if signal == :INT || signal == :TERM
277
+ @pids.dup
278
+ end
279
+ child_pids.each do |pid|
280
+ send_signal_to_pid(signal, pid)
281
+ end
282
+ end
283
+
284
+ def send_signal_to_pid(signal, pid)
285
+ signal = signal.to_sym
286
+ begin
287
+ log_info("Sending signal #{signal} to sidekiq process #{pid}")
288
+ ::Process.kill(signal, pid)
289
+ if [:TERM, :INT].include?(signal)
290
+ Thread.new do
291
+ Thread.current.name = "pid-#{pid}-killer"
292
+ ensure_pid_dies(pid)
293
+ end
245
294
  end
295
+ rescue Errno::ESRCH
296
+ # The process is already dead
246
297
  end
247
- wait_for_children_to_exit(pids) if @process_count == 0
248
298
  end
249
299
 
250
300
  def start_memory_monitor
251
- log_info("Starting memory monitor with max memory #{(@max_memory / (1024**2)).round}mb")
301
+ log_info("Starting memory monitor with max memory #{(@max_memory / (1024**2)).round}mb checked every #{@memory_check_interval} seconds")
252
302
 
253
303
  @memory_monitor = Thread.new do
254
304
  Thread.current.name = "memory-monitor"
@@ -256,20 +306,14 @@ module Sidekiq
256
306
  sleep(@memory_check_interval)
257
307
 
258
308
  pids.each do |pid|
259
- begin
260
- memory = GetProcessMem.new(pid)
261
- if memory.bytes > @max_memory
262
- log_warning("Kill bloated sidekiq process #{pid}: #{memory.mb.round}mb used")
263
- begin
264
- ::Process.kill(:TERM, pid)
265
- rescue Errno::ESRCH
266
- # The process is already dead
267
- end
268
- break
269
- end
270
- rescue => e
271
- log_warning("Error monitoring memory for sidekiq process #{pid}: #{e.inspect}")
309
+ memory = GetProcessMem.new(pid)
310
+ if memory.bytes > @max_memory
311
+ log_warning("Killing bloated sidekiq process #{pid}: #{memory.mb.round}mb used")
312
+ send_signal_to_pid(:TERM, pid)
313
+ break
272
314
  end
315
+ rescue => e
316
+ log_warning("Error monitoring memory for sidekiq process #{pid}: #{e.inspect}")
273
317
  end
274
318
  end
275
319
  end
@@ -281,49 +325,122 @@ module Sidekiq
281
325
  end
282
326
  end
283
327
 
284
- def wait_for_children_to_exit(pids)
285
- timeout = monotonic_time + (sidekiq_options[:timeout] || 25).to_f
286
- pids.each do |pid|
287
- while monotonic_time < timeout
288
- break unless process_alive?(pid)
289
- sleep(0.01)
328
+ def wait_for_children_to_exit
329
+ # Allow the SIGKILL failsafe in ensure_pid_dies time to fire before giving up on a process.
330
+ remaining = reap_child_processes(pids, monotonic_time + sidekiq_shutdown_timeout + @exit_grace_period)
331
+
332
+ unless remaining.empty?
333
+ # Don't leave orphaned processes behind. These pids were never reaped,
334
+ # so the operating system cannot have reused them for another process
335
+ # and it is safe to signal them.
336
+ remaining.each do |pid|
337
+ ::Process.kill(:KILL, pid)
338
+ log_warning("Sidekiq process #{pid} failed to exit; killed with SIGKILL")
339
+ rescue Errno::ESRCH
340
+ # The process is already dead
290
341
  end
342
+
343
+ # SIGKILL cannot be caught or ignored, so the processes will exit, but
344
+ # not necessarily by the time the signal is sent. Wait for them so they
345
+ # are not left behind as zombies.
346
+ remaining = reap_child_processes(remaining, monotonic_time + @exit_grace_period)
291
347
  end
292
348
 
293
- pids.each do |pid|
294
- begin
295
- ::Process.kill(:INT, pid) if process_alive?(pid)
296
- rescue
297
- # Ignore errors so we can continue to kill other processes.
298
- end
349
+ if remaining.empty?
350
+ log_info("All sidekiq processes have exited")
351
+ else
352
+ log_warning("Gave up waiting on sidekiq processes: #{remaining.join(", ")}")
299
353
  end
300
354
  end
301
355
 
302
- def process_alive?(pid)
303
- begin
304
- ::Process.getpgid(pid)
305
- true
306
- rescue Errno::ESRCH
307
- false
356
+ # Reap child processes as they exit until they have all been reaped or the
357
+ # deadline passes.
358
+ #
359
+ # @return [Array<Integer>] The pids that were not reaped.
360
+ def reap_child_processes(child_pids, deadline)
361
+ remaining = child_pids.dup
362
+ until remaining.empty?
363
+ # Poll all of the processes on each pass so that one slow process
364
+ # cannot consume the entire timeout waiting for the others.
365
+ remaining.delete_if { |pid| reap_child_process(pid) }
366
+ break if remaining.empty? || monotonic_time >= deadline
367
+ sleep(0.01)
368
+ end
369
+ remaining
370
+ end
371
+
372
+ # Reap a child process if it has exited. The pid is only removed from the
373
+ # list of running processes when it is actually reaped so that the SIGKILL
374
+ # failsafe in ensure_pid_dies still sees it as running until then.
375
+ #
376
+ # @return [Boolean] true if the process has been reaped or is no longer a child process.
377
+ def reap_child_process(pid)
378
+ reaped = begin
379
+ # The monitoring loop may no longer be running, so reap the child
380
+ # process here rather than just polling for it to disappear.
381
+ ::Process.waitpid(pid, ::Process::WNOHANG)
382
+ rescue Errno::ECHILD, Errno::ESRCH
383
+ # The process has already been reaped.
384
+ pid
385
+ end
386
+ return false unless reaped
387
+
388
+ @mutex.synchronize { @pids.delete(pid) }
389
+ true
390
+ end
391
+
392
+ def sidekiq_shutdown_timeout
393
+ (sidekiq_options[:timeout] || 25).to_f
394
+ end
395
+
396
+ def ensure_pid_dies(pid)
397
+ # Wait for the process to die, or kill it after a timeout. A pid is
398
+ # removed from the list only once it has been reaped, so the list is the
399
+ # source of truth for liveness; a pid still in it cannot have been reused
400
+ # by the operating system for a different process. Checking with a system
401
+ # call is not reliable since it cannot distinguish a recycled pid.
402
+ end_time = monotonic_time + sidekiq_shutdown_timeout
403
+ while monotonic_time < end_time && pids.include?(pid)
404
+ sleep(0.1)
405
+ end
406
+
407
+ if pids.include?(pid)
408
+ begin
409
+ ::Process.kill(:KILL, pid)
410
+ log_warning("Sidekiq process #{pid} failed to exit after #{sidekiq_shutdown_timeout} seconds; killed with SIGKILL")
411
+ rescue Errno::ESRCH
412
+ # The process is already dead
413
+ end
308
414
  end
309
415
  end
310
416
 
311
417
  # Listen for child processes dying and restart if necessary.
312
418
  def monitor_child_processes
313
419
  loop do
314
- pid = ::Process.wait
315
- @mutex.synchronize { @pids.delete(pid) }
420
+ begin
421
+ pid = ::Process.wait
422
+ rescue Errno::ECHILD
423
+ # All child processes are already gone.
424
+ @mutex.synchronize { @pids.clear }
425
+ break
426
+ end
427
+
428
+ # Process.wait reaps any child process, including ones spawned by
429
+ # libraries or application code (i.e. get_process_mem shelling out
430
+ # to ps). Only act on processes forked by the manager.
431
+ removed = @mutex.synchronize { @pids.delete(pid) }
432
+ next unless removed
316
433
 
317
434
  log_info("Sidekiq process #{pid} exited")
318
435
 
319
436
  # If there are not enough processes running, start a replacement one.
320
- if @process_count > @pids.size
437
+ if @process_count > pids.size
321
438
  start_child_process!
322
439
  end
323
440
 
324
441
  set_program_name!
325
442
 
326
- if @pids.empty?
443
+ if pids.empty?
327
444
  break
328
445
  end
329
446
  end
@@ -33,6 +33,31 @@ module Sidekiq
33
33
  end
34
34
  @after_fork = nil
35
35
  end
36
+
37
+ # Parse a max memory value into a number of bytes. The value can be a plain
38
+ # number of bytes or a number suffixed with "m" for megabytes or "g" for
39
+ # gigabytes (case insensitive).
40
+ #
41
+ # @param value [String, nil] The value to parse.
42
+ # @return [Float, nil] The number of bytes or nil if the value is blank.
43
+ # @raise [ArgumentError] If the value cannot be parsed.
44
+ def parse_max_memory(value)
45
+ value = value.to_s.strip
46
+ return nil if value.empty?
47
+
48
+ matched = value.match(/\A(\d+(?:\.\d+)?)([mg]?)\z/i)
49
+ raise ArgumentError.new("Invalid max memory value: #{value.inspect}") unless matched
50
+
51
+ bytes = matched[1].to_f
52
+ case matched[2].downcase
53
+ when "m"
54
+ bytes *= 1024**2
55
+ when "g"
56
+ bytes *= 1024**3
57
+ end
58
+
59
+ bytes
60
+ end
36
61
  end
37
62
  end
38
63
  end
@@ -7,9 +7,16 @@ Gem::Specification.new do |spec|
7
7
  spec.email = ["bbdurand@gmail.com"]
8
8
 
9
9
  spec.summary = "Process manager for forking and monitoring multiple sidekiq processes."
10
+
10
11
  spec.homepage = "https://github.com/bdurand/sidekiq-process_manager"
11
12
  spec.license = "MIT"
12
13
 
14
+ spec.metadata = {
15
+ "homepage_uri" => spec.homepage,
16
+ "source_code_uri" => spec.homepage,
17
+ "changelog_uri" => "#{spec.homepage}/blob/main/CHANGELOG.md"
18
+ }
19
+
13
20
  # Specify which files should be added to the gem when it is released.
14
21
  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
15
22
  ignore_files = %w[
@@ -20,6 +27,7 @@ Gem::Specification.new do |spec|
20
27
  Rakefile
21
28
  gemfiles/
22
29
  spec/
30
+ test_app/
23
31
  ]
24
32
  spec.files = Dir.chdir(File.expand_path("..", __FILE__)) do
25
33
  `git ls-files -z`.split("\x0").reject { |f| ignore_files.any? { |path| f.start_with?(path) } }
@@ -33,6 +41,4 @@ Gem::Specification.new do |spec|
33
41
 
34
42
  spec.add_dependency "sidekiq", ">= 5.0"
35
43
  spec.add_dependency "get_process_mem"
36
-
37
- spec.add_development_dependency "bundler"
38
44
  end
metadata CHANGED
@@ -1,14 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sidekiq-process_manager
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.1.1
4
+ version: 1.1.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Brian Durand
8
- autorequire:
9
8
  bindir: bin
10
9
  cert_chain: []
11
- date: 2023-07-19 00:00:00.000000000 Z
10
+ date: 1980-01-02 00:00:00.000000000 Z
12
11
  dependencies:
13
12
  - !ruby/object:Gem::Dependency
14
13
  name: sidekiq
@@ -38,21 +37,6 @@ dependencies:
38
37
  - - ">="
39
38
  - !ruby/object:Gem::Version
40
39
  version: '0'
41
- - !ruby/object:Gem::Dependency
42
- name: bundler
43
- requirement: !ruby/object:Gem::Requirement
44
- requirements:
45
- - - ">="
46
- - !ruby/object:Gem::Version
47
- version: '0'
48
- type: :development
49
- prerelease: false
50
- version_requirements: !ruby/object:Gem::Requirement
51
- requirements:
52
- - - ">="
53
- - !ruby/object:Gem::Version
54
- version: '0'
55
- description:
56
40
  email:
57
41
  - bbdurand@gmail.com
58
42
  executables:
@@ -73,8 +57,10 @@ files:
73
57
  homepage: https://github.com/bdurand/sidekiq-process_manager
74
58
  licenses:
75
59
  - MIT
76
- metadata: {}
77
- post_install_message:
60
+ metadata:
61
+ homepage_uri: https://github.com/bdurand/sidekiq-process_manager
62
+ source_code_uri: https://github.com/bdurand/sidekiq-process_manager
63
+ changelog_uri: https://github.com/bdurand/sidekiq-process_manager/blob/main/CHANGELOG.md
78
64
  rdoc_options: []
79
65
  require_paths:
80
66
  - lib
@@ -89,8 +75,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
89
75
  - !ruby/object:Gem::Version
90
76
  version: '0'
91
77
  requirements: []
92
- rubygems_version: 3.4.12
93
- signing_key:
78
+ rubygems_version: 4.0.3
94
79
  specification_version: 4
95
80
  summary: Process manager for forking and monitoring multiple sidekiq processes.
96
81
  test_files: []