sidekiq-process_manager 1.1.2 → 1.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +15 -0
- data/README.md +27 -1
- data/VERSION +1 -1
- data/bin/sidekiq-process-manager +21 -12
- data/lib/sidekiq/process_manager/manager.rb +142 -44
- data/lib/sidekiq/process_manager.rb +25 -0
- data/sidekiq-process_manager.gemspec +8 -2
- metadata +7 -22
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 24bc40ee7ef3f0ade56a7c116d509cc7cbb88c4f15c1781a6c065be88ebe4a0c
|
|
4
|
+
data.tar.gz: 9c3704f538d9cb990aa4cb55e4fc2ce8f4b59c60aa0de92d0a95f1cca44e30fc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fe31823918abde6d3128ada00c22aa60e25e3d16c7a3ebe85fbeaa25d20aa228ad5aee0ed68c0746876ae98bebbc951ea46725c0a339d583566befa82a2d1b1c
|
|
7
|
+
data.tar.gz: 39f00a80aaef0fc0e936f592a8b592a497ba697c366787d006924e1df73239db424a74d354bada8fadebb6f51e8bea87c44b4807b5b56a994a29902a2930526c
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,21 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## 1.1.3
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
- The interval between memory checks can now be set with the `--memory-check-interval` command line argument or the `SIDEKIQ_MEMORY_CHECK_INTERVAL` environment variable. It still defaults to 60 seconds.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
- Child processes now reset the signal handlers inherited from the process manager to their default behavior when they are forked. The inherited handlers do nothing outside of the master process, so any signal sent to a child process before sidekiq installed its own handlers was silently discarded and the process would only be stopped by the SIGKILL failsafe.
|
|
14
|
+
- Fixed race condition where a child process forked to replace a dead process while a TERM or INT signal was being handled would never receive the shutdown signal, leaving the process manager hung waiting on it.
|
|
15
|
+
- Child processes are now reaped while waiting for them to exit at shutdown, with a timeout, so the process manager can no longer hang forever waiting on zombie processes if it exits abnormally. All processes are polled on each pass so that one slow process cannot consume the entire timeout, and any that are still running when the timeout expires are killed with SIGKILL and reaped rather than being orphaned.
|
|
16
|
+
- Child processes spawned by libraries or application code (e.g. `ps` invoked by the get_process_mem gem) are no longer mistaken for sidekiq processes by the process monitor.
|
|
17
|
+
- The SIGKILL failsafe now uses the manager's list of running processes to determine liveness instead of a system call, removing the possibility of signaling an unrelated process if the operating system reuses a process id.
|
|
18
|
+
- The signal handling thread exits cleanly instead of busy-looping if the signal pipe is closed.
|
|
19
|
+
- Fixed the validation error message when the process count is less than one.
|
|
20
|
+
- The `sidekiq-process-manager` command line now accepts a max memory value in plain bytes and aborts with an error message on invalid values instead of silently ignoring them.
|
|
21
|
+
|
|
7
22
|
## 1.1.2
|
|
8
23
|
|
|
9
24
|
### Added
|
data/README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Sidekiq::ProcessManager
|
|
2
2
|
|
|
3
3
|
[](https://github.com/bdurand/sidekiq-process_manager/actions/workflows/continuous_integration.yml)
|
|
4
|
-
[](https://codeclimate.com/github/bdurand/sidekiq-process_manager/maintainability)
|
|
5
4
|
[](https://github.com/testdouble/standard)
|
|
5
|
+
[](https://badge.fury.io/rb/sidekiq-process_manager)
|
|
6
6
|
|
|
7
7
|
This gem provides a command line script for managing [sidekiq](https://github.com/mperham/sidekiq) processes. It starts up a process that then forks multiple sidekiq processes and manages their life cycle. This is important for large sidekiq installations, since without it on MRI ruby, sidekiq will only use one CPU core. By starting multiple processes you make all cores available.
|
|
8
8
|
|
|
@@ -109,6 +109,18 @@ or
|
|
|
109
109
|
SIDEKIQ_MAX_MEMORY=2000m SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
|
|
110
110
|
```
|
|
111
111
|
|
|
112
|
+
Memory is checked once a minute by default. You can change how often with the `--memory-check-interval` argument or with the `SIDEKIQ_MEMORY_CHECK_INTERVAL` environment variable. The value is in seconds. Note that only one process is restarted per check, so the interval also controls how quickly multiple bloated processes are recycled.
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
bundle exec sidekiq-process-manager --processes 4 --max-memory 2g --memory-check-interval 10
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
or
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
SIDEKIQ_MEMORY_CHECK_INTERVAL=10 SIDEKIQ_MAX_MEMORY=2g SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
|
|
122
|
+
```
|
|
123
|
+
|
|
112
124
|
## Alternatives
|
|
113
125
|
|
|
114
126
|
Any process manager can be an alternative (service, update, systemd, monit, god, etc.).
|
|
@@ -139,6 +151,20 @@ Or install it yourself as:
|
|
|
139
151
|
$ gem install sidekiq-process_manager
|
|
140
152
|
```
|
|
141
153
|
|
|
154
|
+
## Test Application
|
|
155
|
+
|
|
156
|
+
There is a test application in the `test_app` directory that can be used to test the process manager. You can run it with:
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
$ cd test_app
|
|
160
|
+
$ bundle install
|
|
161
|
+
$ bin/run
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
This will bring up a valkey docker instance, sidekiq cluster, and rack application that shows the status of the sidekiq processes.
|
|
165
|
+
|
|
166
|
+
Then visit [http://localhost:9292](http://localhost:9292) to see the status of the sidekiq processes.
|
|
167
|
+
|
|
142
168
|
## Contributing
|
|
143
169
|
|
|
144
170
|
Open a pull request on [GitHub](https://github.com/bdurand/sidekiq-process_manager).
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
1.1.
|
|
1
|
+
1.1.3
|
data/bin/sidekiq-process-manager
CHANGED
|
@@ -7,19 +7,23 @@ require_relative "../lib/sidekiq-process_manager"
|
|
|
7
7
|
DEFAULT_PROCESS_COUNT = 1
|
|
8
8
|
|
|
9
9
|
def parse_max_memory(max_memory)
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
10
|
+
Sidekiq::ProcessManager.parse_max_memory(max_memory)
|
|
11
|
+
rescue ArgumentError => e
|
|
12
|
+
abort(e.message)
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def parse_memory_check_interval(interval)
|
|
16
|
+
interval = interval.to_s.strip
|
|
17
|
+
return nil if interval.empty?
|
|
18
|
+
|
|
19
|
+
begin
|
|
20
|
+
interval = Float(interval)
|
|
21
|
+
rescue ArgumentError, TypeError
|
|
22
|
+
abort("Invalid memory check interval value: #{interval.inspect}")
|
|
20
23
|
end
|
|
24
|
+
abort("Memory check interval must be greater than 0") unless interval > 0
|
|
21
25
|
|
|
22
|
-
|
|
26
|
+
interval
|
|
23
27
|
end
|
|
24
28
|
|
|
25
29
|
options = {
|
|
@@ -27,6 +31,7 @@ options = {
|
|
|
27
31
|
prefork: !ENV.fetch("SIDEKIQ_PREFORK", "").empty?,
|
|
28
32
|
preboot: ENV["SIDEKIQ_PREBOOT"],
|
|
29
33
|
max_memory: parse_max_memory(ENV["SIDEKIQ_MAX_MEMORY"]),
|
|
34
|
+
memory_check_interval: parse_memory_check_interval(ENV["SIDEKIQ_MEMORY_CHECK_INTERVAL"]),
|
|
30
35
|
mode: nil,
|
|
31
36
|
}
|
|
32
37
|
|
|
@@ -45,10 +50,14 @@ parser = OptionParser.new do |opts|
|
|
|
45
50
|
options[:preboot] = preboot
|
|
46
51
|
end
|
|
47
52
|
|
|
48
|
-
opts.on('--max-memory MEMORY', "Max memory for each process (can also specify with SIDEKIQ_MAX_MEMORY); suffix with m or g to specify megabytes or gigabytes") do |max_memory|
|
|
53
|
+
opts.on('--max-memory MEMORY', "Max memory for each process (can also specify with SIDEKIQ_MAX_MEMORY); value is in bytes, or suffix with m or g to specify megabytes or gigabytes") do |max_memory|
|
|
49
54
|
options[:max_memory] = parse_max_memory(max_memory)
|
|
50
55
|
end
|
|
51
56
|
|
|
57
|
+
opts.on('--memory-check-interval SECONDS', "Seconds between memory checks when a max memory is set (can also specify with SIDEKIQ_MEMORY_CHECK_INTERVAL); defaults to #{Sidekiq::ProcessManager::Manager::DEFAULT_MEMORY_CHECK_INTERVAL} seconds") do |interval|
|
|
58
|
+
options[:memory_check_interval] = parse_memory_check_interval(interval)
|
|
59
|
+
end
|
|
60
|
+
|
|
52
61
|
opts.on('--testing', "Enable test mode") do |testing|
|
|
53
62
|
options[:mode] = :testing if testing
|
|
54
63
|
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "sidekiq"
|
|
4
4
|
require "get_process_mem"
|
|
5
|
+
require "timeout"
|
|
5
6
|
|
|
6
7
|
module Sidekiq
|
|
7
8
|
module ProcessManager
|
|
@@ -9,6 +10,12 @@ module Sidekiq
|
|
|
9
10
|
# that the specified number of sidekiq processes are running. It will also forward
|
|
10
11
|
# signals sent to the main process to the child processes.
|
|
11
12
|
class Manager
|
|
13
|
+
# Signals that are trapped by the process manager and forwarded to the child processes.
|
|
14
|
+
SIGNALS = [:INT, :TERM, :USR1, :USR2, :TSTP, :TTIN].freeze
|
|
15
|
+
|
|
16
|
+
# Number of seconds between memory checks when a max memory is set.
|
|
17
|
+
DEFAULT_MEMORY_CHECK_INTERVAL = 60
|
|
18
|
+
|
|
12
19
|
attr_reader :cli
|
|
13
20
|
|
|
14
21
|
# Create a new process manager.
|
|
@@ -16,31 +23,38 @@ module Sidekiq
|
|
|
16
23
|
# @param process_count [Integer] The number of sidekiq processes to start.
|
|
17
24
|
# @param prefork [Boolean] If true, the process manager will load the application before forking.
|
|
18
25
|
# @param preboot [String] If set, the process manager will require the specified file before forking the child processes.
|
|
26
|
+
# @param memory_check_interval [Numeric] The number of seconds between memory checks
|
|
27
|
+
# when a max memory is set.
|
|
19
28
|
# @param mode [Symbol] If set to :testing, the process manager will use a mock CLI.
|
|
20
29
|
# @param silent [Boolean] If true, the process manager will not output any messages.
|
|
21
|
-
def initialize(process_count: 1, prefork: false, preboot: nil, max_memory: nil, mode: nil, silent: false)
|
|
30
|
+
def initialize(process_count: 1, prefork: false, preboot: nil, max_memory: nil, memory_check_interval: nil, mode: nil, silent: false)
|
|
22
31
|
require "sidekiq/cli"
|
|
23
32
|
|
|
24
33
|
# Get the number of processes to fork
|
|
25
34
|
@process_count = process_count
|
|
26
|
-
raise ArgumentError.new("Process count must be greater than
|
|
35
|
+
raise ArgumentError.new("Process count must be greater than 0") if @process_count < 1
|
|
27
36
|
|
|
28
|
-
@prefork =
|
|
37
|
+
@prefork = prefork && process_count > 1
|
|
29
38
|
@preboot = preboot if process_count > 1 && !prefork
|
|
30
39
|
@max_memory = ((max_memory.to_i > 0) ? max_memory.to_i : nil)
|
|
31
40
|
|
|
41
|
+
if !memory_check_interval.nil? && memory_check_interval.to_f <= 0
|
|
42
|
+
raise ArgumentError.new("Memory check interval must be greater than 0")
|
|
43
|
+
end
|
|
44
|
+
|
|
32
45
|
if mode == :testing
|
|
33
46
|
require_relative "../../../spec/support/mocks"
|
|
34
47
|
@cli = MockSidekiqCLI.new(silent)
|
|
35
|
-
@memory_check_interval = 1
|
|
48
|
+
@memory_check_interval = (memory_check_interval || 1).to_f
|
|
49
|
+
@exit_grace_period = 0.5
|
|
36
50
|
else
|
|
37
51
|
@cli = Sidekiq::CLI.instance
|
|
38
|
-
@memory_check_interval =
|
|
52
|
+
@memory_check_interval = (memory_check_interval || DEFAULT_MEMORY_CHECK_INTERVAL).to_f
|
|
53
|
+
@exit_grace_period = 5
|
|
39
54
|
end
|
|
40
55
|
|
|
41
56
|
@silent = silent
|
|
42
57
|
@pids = []
|
|
43
|
-
@terminated_pids = []
|
|
44
58
|
@started = false
|
|
45
59
|
@mutex = Mutex.new
|
|
46
60
|
end
|
|
@@ -69,7 +83,10 @@ module Sidekiq
|
|
|
69
83
|
|
|
70
84
|
while @signal_pipe_read.wait_readable
|
|
71
85
|
begin
|
|
72
|
-
signal = @signal_pipe_read.gets
|
|
86
|
+
signal = @signal_pipe_read.gets
|
|
87
|
+
break if signal.nil?
|
|
88
|
+
|
|
89
|
+
signal = signal.strip
|
|
73
90
|
send_signal_to_children(signal.to_sym)
|
|
74
91
|
rescue => e
|
|
75
92
|
log_warning("Error sending signal #{signal} to child processes: #{e.message}")
|
|
@@ -78,7 +95,7 @@ module Sidekiq
|
|
|
78
95
|
end
|
|
79
96
|
|
|
80
97
|
# Trap signals that will be forwarded to child processes
|
|
81
|
-
|
|
98
|
+
SIGNALS.each do |signal|
|
|
82
99
|
::Signal.trap(signal) do
|
|
83
100
|
if ::Process.pid == master_pid
|
|
84
101
|
signal = :TERM if signal == :INT
|
|
@@ -101,7 +118,7 @@ module Sidekiq
|
|
|
101
118
|
GC.start
|
|
102
119
|
GC.compact if GC.respond_to?(:compact)
|
|
103
120
|
# I'm not sure why, but running GC operations blocks until we try to write some I/O.
|
|
104
|
-
File.write(
|
|
121
|
+
File.write(File::NULL, "0")
|
|
105
122
|
|
|
106
123
|
@process_count.times do
|
|
107
124
|
start_child_process!
|
|
@@ -120,7 +137,7 @@ module Sidekiq
|
|
|
120
137
|
def wait(timeout = 5)
|
|
121
138
|
timeout_time = monotonic_time + timeout
|
|
122
139
|
while monotonic_time <= timeout_time
|
|
123
|
-
return if
|
|
140
|
+
return if pids.size == @process_count
|
|
124
141
|
sleep(0.01)
|
|
125
142
|
end
|
|
126
143
|
|
|
@@ -217,11 +234,20 @@ module Sidekiq
|
|
|
217
234
|
end
|
|
218
235
|
|
|
219
236
|
def set_program_name!
|
|
220
|
-
$PROGRAM_NAME = "sidekiq process manager #{sidekiq_options[:tag]} [#{
|
|
237
|
+
$PROGRAM_NAME = "sidekiq process manager #{sidekiq_options[:tag]} [#{pids.size} processes]"
|
|
221
238
|
end
|
|
222
239
|
|
|
223
240
|
def start_child_process!
|
|
224
241
|
pid = fork do
|
|
242
|
+
# Child processes inherit the signal handlers installed by the process
|
|
243
|
+
# manager. Those handlers only do anything in the master process, so
|
|
244
|
+
# leaving them in place would silently discard any signal sent to the
|
|
245
|
+
# child before sidekiq installs its own handlers. Restore the default
|
|
246
|
+
# behavior until then so signals are never swallowed.
|
|
247
|
+
SIGNALS.each do |signal|
|
|
248
|
+
::Signal.trap(signal, "DEFAULT")
|
|
249
|
+
end
|
|
250
|
+
|
|
225
251
|
# Set $0 so instrumentation libraries detecting sidekiq from the command run will work properly.
|
|
226
252
|
$0 = File.join(File.dirname($0), "sidekiq")
|
|
227
253
|
@process_count = 0
|
|
@@ -232,15 +258,25 @@ module Sidekiq
|
|
|
232
258
|
Sidekiq::ProcessManager.run_after_fork_hooks
|
|
233
259
|
@cli.run
|
|
234
260
|
end
|
|
235
|
-
|
|
261
|
+
shutting_down = false
|
|
262
|
+
@mutex.synchronize do
|
|
263
|
+
@pids << pid
|
|
264
|
+
shutting_down = @process_count < 1
|
|
265
|
+
end
|
|
236
266
|
log_info("Forked sidekiq process with pid #{pid}")
|
|
267
|
+
# A shutdown signal may have been forwarded to the other processes while
|
|
268
|
+
# this one was being forked; it needs to get the signal as well.
|
|
269
|
+
send_signal_to_pid(:TERM, pid) if shutting_down
|
|
237
270
|
set_program_name!
|
|
238
271
|
end
|
|
239
272
|
|
|
240
273
|
def send_signal_to_children(signal)
|
|
241
274
|
log_info("Process manager trapped signal #{signal}")
|
|
242
|
-
|
|
243
|
-
|
|
275
|
+
child_pids = @mutex.synchronize do
|
|
276
|
+
@process_count = 0 if signal == :INT || signal == :TERM
|
|
277
|
+
@pids.dup
|
|
278
|
+
end
|
|
279
|
+
child_pids.each do |pid|
|
|
244
280
|
send_signal_to_pid(signal, pid)
|
|
245
281
|
end
|
|
246
282
|
end
|
|
@@ -262,7 +298,7 @@ module Sidekiq
|
|
|
262
298
|
end
|
|
263
299
|
|
|
264
300
|
def start_memory_monitor
|
|
265
|
-
log_info("Starting memory monitor with max memory #{(@max_memory / (1024**2)).round}mb")
|
|
301
|
+
log_info("Starting memory monitor with max memory #{(@max_memory / (1024**2)).round}mb checked every #{@memory_check_interval} seconds")
|
|
266
302
|
|
|
267
303
|
@memory_monitor = Thread.new do
|
|
268
304
|
Thread.current.name = "memory-monitor"
|
|
@@ -270,16 +306,14 @@ module Sidekiq
|
|
|
270
306
|
sleep(@memory_check_interval)
|
|
271
307
|
|
|
272
308
|
pids.each do |pid|
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
break
|
|
279
|
-
end
|
|
280
|
-
rescue => e
|
|
281
|
-
log_warning("Error monitoring memory for sidekiq process #{pid}: #{e.inspect}")
|
|
309
|
+
memory = GetProcessMem.new(pid)
|
|
310
|
+
if memory.bytes > @max_memory
|
|
311
|
+
log_warning("Killing bloated sidekiq process #{pid}: #{memory.mb.round}mb used")
|
|
312
|
+
send_signal_to_pid(:TERM, pid)
|
|
313
|
+
break
|
|
282
314
|
end
|
|
315
|
+
rescue => e
|
|
316
|
+
log_warning("Error monitoring memory for sidekiq process #{pid}: #{e.inspect}")
|
|
283
317
|
end
|
|
284
318
|
end
|
|
285
319
|
end
|
|
@@ -292,35 +326,88 @@ module Sidekiq
|
|
|
292
326
|
end
|
|
293
327
|
|
|
294
328
|
def wait_for_children_to_exit
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
329
|
+
# Allow the SIGKILL failsafe in ensure_pid_dies time to fire before giving up on a process.
|
|
330
|
+
remaining = reap_child_processes(pids, monotonic_time + sidekiq_shutdown_timeout + @exit_grace_period)
|
|
331
|
+
|
|
332
|
+
unless remaining.empty?
|
|
333
|
+
# Don't leave orphaned processes behind. These pids were never reaped,
|
|
334
|
+
# so the operating system cannot have reused them for another process
|
|
335
|
+
# and it is safe to signal them.
|
|
336
|
+
remaining.each do |pid|
|
|
337
|
+
::Process.kill(:KILL, pid)
|
|
338
|
+
log_warning("Sidekiq process #{pid} failed to exit; killed with SIGKILL")
|
|
339
|
+
rescue Errno::ESRCH
|
|
340
|
+
# The process is already dead
|
|
298
341
|
end
|
|
342
|
+
|
|
343
|
+
# SIGKILL cannot be caught or ignored, so the processes will exit, but
|
|
344
|
+
# not necessarily by the time the signal is sent. Wait for them so they
|
|
345
|
+
# are not left behind as zombies.
|
|
346
|
+
remaining = reap_child_processes(remaining, monotonic_time + @exit_grace_period)
|
|
347
|
+
end
|
|
348
|
+
|
|
349
|
+
if remaining.empty?
|
|
350
|
+
log_info("All sidekiq processes have exited")
|
|
351
|
+
else
|
|
352
|
+
log_warning("Gave up waiting on sidekiq processes: #{remaining.join(", ")}")
|
|
299
353
|
end
|
|
300
|
-
log_info("All sidekiq processes have exited")
|
|
301
354
|
end
|
|
302
355
|
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
356
|
+
# Reap child processes as they exit until they have all been reaped or the
|
|
357
|
+
# deadline passes.
|
|
358
|
+
#
|
|
359
|
+
# @return [Array<Integer>] The pids that were not reaped.
|
|
360
|
+
def reap_child_processes(child_pids, deadline)
|
|
361
|
+
remaining = child_pids.dup
|
|
362
|
+
until remaining.empty?
|
|
363
|
+
# Poll all of the processes on each pass so that one slow process
|
|
364
|
+
# cannot consume the entire timeout waiting for the others.
|
|
365
|
+
remaining.delete_if { |pid| reap_child_process(pid) }
|
|
366
|
+
break if remaining.empty? || monotonic_time >= deadline
|
|
367
|
+
sleep(0.01)
|
|
368
|
+
end
|
|
369
|
+
remaining
|
|
370
|
+
end
|
|
371
|
+
|
|
372
|
+
# Reap a child process if it has exited. The pid is only removed from the
|
|
373
|
+
# list of running processes when it is actually reaped so that the SIGKILL
|
|
374
|
+
# failsafe in ensure_pid_dies still sees it as running until then.
|
|
375
|
+
#
|
|
376
|
+
# @return [Boolean] true if the process has been reaped or is no longer a child process.
|
|
377
|
+
def reap_child_process(pid)
|
|
378
|
+
reaped = begin
|
|
379
|
+
# The monitoring loop may no longer be running, so reap the child
|
|
380
|
+
# process here rather than just polling for it to disappear.
|
|
381
|
+
::Process.waitpid(pid, ::Process::WNOHANG)
|
|
382
|
+
rescue Errno::ECHILD, Errno::ESRCH
|
|
383
|
+
# The process has already been reaped.
|
|
384
|
+
pid
|
|
309
385
|
end
|
|
386
|
+
return false unless reaped
|
|
387
|
+
|
|
388
|
+
@mutex.synchronize { @pids.delete(pid) }
|
|
389
|
+
true
|
|
390
|
+
end
|
|
391
|
+
|
|
392
|
+
def sidekiq_shutdown_timeout
|
|
393
|
+
(sidekiq_options[:timeout] || 25).to_f
|
|
310
394
|
end
|
|
311
395
|
|
|
312
396
|
def ensure_pid_dies(pid)
|
|
313
|
-
# Wait for the process to die, or kill it after a timeout.
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
397
|
+
# Wait for the process to die, or kill it after a timeout. A pid is
|
|
398
|
+
# removed from the list only once it has been reaped, so the list is the
|
|
399
|
+
# source of truth for liveness; a pid still in it cannot have been reused
|
|
400
|
+
# by the operating system for a different process. Checking with a system
|
|
401
|
+
# call is not reliable since it cannot distinguish a recycled pid.
|
|
402
|
+
end_time = monotonic_time + sidekiq_shutdown_timeout
|
|
403
|
+
while monotonic_time < end_time && pids.include?(pid)
|
|
317
404
|
sleep(0.1)
|
|
318
405
|
end
|
|
319
406
|
|
|
320
|
-
if
|
|
407
|
+
if pids.include?(pid)
|
|
321
408
|
begin
|
|
322
409
|
::Process.kill(:KILL, pid)
|
|
323
|
-
log_warning("Sidekiq process #{pid} failed to exit after #{
|
|
410
|
+
log_warning("Sidekiq process #{pid} failed to exit after #{sidekiq_shutdown_timeout} seconds; killed with SIGKILL")
|
|
324
411
|
rescue Errno::ESRCH
|
|
325
412
|
# The process is already dead
|
|
326
413
|
end
|
|
@@ -330,19 +417,30 @@ module Sidekiq
|
|
|
330
417
|
# Listen for child processes dying and restart if necessary.
|
|
331
418
|
def monitor_child_processes
|
|
332
419
|
loop do
|
|
333
|
-
|
|
334
|
-
|
|
420
|
+
begin
|
|
421
|
+
pid = ::Process.wait
|
|
422
|
+
rescue Errno::ECHILD
|
|
423
|
+
# All child processes are already gone.
|
|
424
|
+
@mutex.synchronize { @pids.clear }
|
|
425
|
+
break
|
|
426
|
+
end
|
|
427
|
+
|
|
428
|
+
# Process.wait reaps any child process, including ones spawned by
|
|
429
|
+
# libraries or application code (i.e. get_process_mem shelling out
|
|
430
|
+
# to ps). Only act on processes forked by the manager.
|
|
431
|
+
removed = @mutex.synchronize { @pids.delete(pid) }
|
|
432
|
+
next unless removed
|
|
335
433
|
|
|
336
434
|
log_info("Sidekiq process #{pid} exited")
|
|
337
435
|
|
|
338
436
|
# If there are not enough processes running, start a replacement one.
|
|
339
|
-
if @process_count >
|
|
437
|
+
if @process_count > pids.size
|
|
340
438
|
start_child_process!
|
|
341
439
|
end
|
|
342
440
|
|
|
343
441
|
set_program_name!
|
|
344
442
|
|
|
345
|
-
if
|
|
443
|
+
if pids.empty?
|
|
346
444
|
break
|
|
347
445
|
end
|
|
348
446
|
end
|
|
@@ -33,6 +33,31 @@ module Sidekiq
|
|
|
33
33
|
end
|
|
34
34
|
@after_fork = nil
|
|
35
35
|
end
|
|
36
|
+
|
|
37
|
+
# Parse a max memory value into a number of bytes. The value can be a plain
|
|
38
|
+
# number of bytes or a number suffixed with "m" for megabytes or "g" for
|
|
39
|
+
# gigabytes (case insensitive).
|
|
40
|
+
#
|
|
41
|
+
# @param value [String, nil] The value to parse.
|
|
42
|
+
# @return [Float, nil] The number of bytes or nil if the value is blank.
|
|
43
|
+
# @raise [ArgumentError] If the value cannot be parsed.
|
|
44
|
+
def parse_max_memory(value)
|
|
45
|
+
value = value.to_s.strip
|
|
46
|
+
return nil if value.empty?
|
|
47
|
+
|
|
48
|
+
matched = value.match(/\A(\d+(?:\.\d+)?)([mg]?)\z/i)
|
|
49
|
+
raise ArgumentError.new("Invalid max memory value: #{value.inspect}") unless matched
|
|
50
|
+
|
|
51
|
+
bytes = matched[1].to_f
|
|
52
|
+
case matched[2].downcase
|
|
53
|
+
when "m"
|
|
54
|
+
bytes *= 1024**2
|
|
55
|
+
when "g"
|
|
56
|
+
bytes *= 1024**3
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
bytes
|
|
60
|
+
end
|
|
36
61
|
end
|
|
37
62
|
end
|
|
38
63
|
end
|
|
@@ -7,9 +7,16 @@ Gem::Specification.new do |spec|
|
|
|
7
7
|
spec.email = ["bbdurand@gmail.com"]
|
|
8
8
|
|
|
9
9
|
spec.summary = "Process manager for forking and monitoring multiple sidekiq processes."
|
|
10
|
+
|
|
10
11
|
spec.homepage = "https://github.com/bdurand/sidekiq-process_manager"
|
|
11
12
|
spec.license = "MIT"
|
|
12
13
|
|
|
14
|
+
spec.metadata = {
|
|
15
|
+
"homepage_uri" => spec.homepage,
|
|
16
|
+
"source_code_uri" => spec.homepage,
|
|
17
|
+
"changelog_uri" => "#{spec.homepage}/blob/main/CHANGELOG.md"
|
|
18
|
+
}
|
|
19
|
+
|
|
13
20
|
# Specify which files should be added to the gem when it is released.
|
|
14
21
|
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
|
15
22
|
ignore_files = %w[
|
|
@@ -20,6 +27,7 @@ Gem::Specification.new do |spec|
|
|
|
20
27
|
Rakefile
|
|
21
28
|
gemfiles/
|
|
22
29
|
spec/
|
|
30
|
+
test_app/
|
|
23
31
|
]
|
|
24
32
|
spec.files = Dir.chdir(File.expand_path("..", __FILE__)) do
|
|
25
33
|
`git ls-files -z`.split("\x0").reject { |f| ignore_files.any? { |path| f.start_with?(path) } }
|
|
@@ -33,6 +41,4 @@ Gem::Specification.new do |spec|
|
|
|
33
41
|
|
|
34
42
|
spec.add_dependency "sidekiq", ">= 5.0"
|
|
35
43
|
spec.add_dependency "get_process_mem"
|
|
36
|
-
|
|
37
|
-
spec.add_development_dependency "bundler"
|
|
38
44
|
end
|
metadata
CHANGED
|
@@ -1,14 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: sidekiq-process_manager
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.1.
|
|
4
|
+
version: 1.1.3
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Brian Durand
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
13
|
name: sidekiq
|
|
@@ -38,21 +37,6 @@ dependencies:
|
|
|
38
37
|
- - ">="
|
|
39
38
|
- !ruby/object:Gem::Version
|
|
40
39
|
version: '0'
|
|
41
|
-
- !ruby/object:Gem::Dependency
|
|
42
|
-
name: bundler
|
|
43
|
-
requirement: !ruby/object:Gem::Requirement
|
|
44
|
-
requirements:
|
|
45
|
-
- - ">="
|
|
46
|
-
- !ruby/object:Gem::Version
|
|
47
|
-
version: '0'
|
|
48
|
-
type: :development
|
|
49
|
-
prerelease: false
|
|
50
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
51
|
-
requirements:
|
|
52
|
-
- - ">="
|
|
53
|
-
- !ruby/object:Gem::Version
|
|
54
|
-
version: '0'
|
|
55
|
-
description:
|
|
56
40
|
email:
|
|
57
41
|
- bbdurand@gmail.com
|
|
58
42
|
executables:
|
|
@@ -73,8 +57,10 @@ files:
|
|
|
73
57
|
homepage: https://github.com/bdurand/sidekiq-process_manager
|
|
74
58
|
licenses:
|
|
75
59
|
- MIT
|
|
76
|
-
metadata:
|
|
77
|
-
|
|
60
|
+
metadata:
|
|
61
|
+
homepage_uri: https://github.com/bdurand/sidekiq-process_manager
|
|
62
|
+
source_code_uri: https://github.com/bdurand/sidekiq-process_manager
|
|
63
|
+
changelog_uri: https://github.com/bdurand/sidekiq-process_manager/blob/main/CHANGELOG.md
|
|
78
64
|
rdoc_options: []
|
|
79
65
|
require_paths:
|
|
80
66
|
- lib
|
|
@@ -89,8 +75,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
89
75
|
- !ruby/object:Gem::Version
|
|
90
76
|
version: '0'
|
|
91
77
|
requirements: []
|
|
92
|
-
rubygems_version:
|
|
93
|
-
signing_key:
|
|
78
|
+
rubygems_version: 4.0.3
|
|
94
79
|
specification_version: 4
|
|
95
80
|
summary: Process manager for forking and monitoring multiple sidekiq processes.
|
|
96
81
|
test_files: []
|