sidekiq-process_manager 1.1.1 → 1.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +23 -0
- data/README.md +29 -3
- data/VERSION +1 -1
- data/bin/sidekiq-process-manager +21 -12
- data/lib/sidekiq/process_manager/manager.rb +178 -61
- data/lib/sidekiq/process_manager.rb +25 -0
- data/sidekiq-process_manager.gemspec +8 -2
- metadata +7 -22
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 24bc40ee7ef3f0ade56a7c116d509cc7cbb88c4f15c1781a6c065be88ebe4a0c
|
|
4
|
+
data.tar.gz: 9c3704f538d9cb990aa4cb55e4fc2ce8f4b59c60aa0de92d0a95f1cca44e30fc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fe31823918abde6d3128ada00c22aa60e25e3d16c7a3ebe85fbeaa25d20aa228ad5aee0ed68c0746876ae98bebbc951ea46725c0a339d583566befa82a2d1b1c
|
|
7
|
+
data.tar.gz: 39f00a80aaef0fc0e936f592a8b592a497ba697c366787d006924e1df73239db424a74d354bada8fadebb6f51e8bea87c44b4807b5b56a994a29902a2930526c
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,29 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## 1.1.3
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
- The interval between memory checks can now be set with the `--memory-check-interval` command line argument or the `SIDEKIQ_MEMORY_CHECK_INTERVAL` environment variable. It still defaults to 60 seconds.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
- Child processes now reset the signal handlers inherited from the process manager to their default behavior when they are forked. The inherited handlers do nothing outside of the master process, so any signal sent to a child process before sidekiq installed its own handlers was silently discarded and the process would only be stopped by the SIGKILL failsafe.
|
|
14
|
+
- Fixed race condition where a child process forked to replace a dead process while a TERM or INT signal was being handled would never receive the shutdown signal, leaving the process manager hung waiting on it.
|
|
15
|
+
- Child processes are now reaped while waiting for them to exit at shutdown, with a timeout, so the process manager can no longer hang forever waiting on zombie processes if it exits abnormally. All processes are polled on each pass so that one slow process cannot consume the entire timeout, and any that are still running when the timeout expires are killed with SIGKILL and reaped rather than being orphaned.
|
|
16
|
+
- Child processes spawned by libraries or application code (e.g. `ps` invoked by the get_process_mem gem) are no longer mistaken for sidekiq processes by the process monitor.
|
|
17
|
+
- The SIGKILL failsafe now uses the manager's list of running processes to determine liveness instead of a system call, removing the possibility of signaling an unrelated process if the operating system reuses a process id.
|
|
18
|
+
- The signal handling thread exits cleanly instead of busy-looping if the signal pipe is closed.
|
|
19
|
+
- Fixed the validation error message when the process count is less than one.
|
|
20
|
+
- The `sidekiq-process-manager` command line now accepts a max memory value in plain bytes and aborts with an error message on invalid values instead of silently ignoring them.
|
|
21
|
+
|
|
22
|
+
## 1.1.2
|
|
23
|
+
|
|
24
|
+
### Added
|
|
25
|
+
- Added thread to monitor child processes to make sure they exit after a SIGTERM signal has been sent. If a process does not exit after the configured Sidekiq timeout time, then it will be killed with a SIGKILL signal.
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
- A SIGINT sent to the manager process will sent SIGTERM to the child processes to give them a chance to shutdown gracefully.
|
|
29
|
+
|
|
7
30
|
## 1.1.1
|
|
8
31
|
|
|
9
32
|
### Added
|
data/README.md
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
# Sidekiq::ProcessManager
|
|
2
2
|
|
|
3
3
|
[](https://github.com/bdurand/sidekiq-process_manager/actions/workflows/continuous_integration.yml)
|
|
4
|
-
[](https://codeclimate.com/github/bdurand/sidekiq-process_manager/maintainability)
|
|
5
4
|
[](https://github.com/testdouble/standard)
|
|
5
|
+
[](https://badge.fury.io/rb/sidekiq-process_manager)
|
|
6
6
|
|
|
7
7
|
This gem provides a command line script for managing [sidekiq](https://github.com/mperham/sidekiq) processes. It starts up a process that then forks multiple sidekiq processes and manages their life cycle. This is important for large sidekiq installations, since without it on MRI ruby, sidekiq will only use one CPU core. By starting multiple processes you make all cores available.
|
|
8
8
|
|
|
9
9
|
The sidekiq processes can all be managed by sending signals to the manager process. This process simply forwards the signals on to the child processes, allowing you to control the sidekiq processes as you normally would.
|
|
10
10
|
|
|
11
|
-
If one of the sidekiq processes
|
|
11
|
+
If one of the sidekiq processes exits unexpectedly, the process manager automatically starts a new sidekiq process to replace it.
|
|
12
12
|
|
|
13
13
|
## Pre-Forking
|
|
14
14
|
|
|
@@ -44,7 +44,7 @@ For a Rails application, you would normally want to preboot the `config/boot.rb`
|
|
|
44
44
|
|
|
45
45
|
## Memory Bloat
|
|
46
46
|
|
|
47
|
-
You can also specify a maximum memory footprint that you want to allow for each child process. You can use this feature to automatically guard against poorly designed workers that bloat the Ruby memory heap. Note that you can also use an external process monitor to kill processes with memory bloat; the process manager will restart any process regardless of how it
|
|
47
|
+
You can also specify a maximum memory footprint that you want to allow for each child process. You can use this feature to automatically guard against poorly designed workers that bloat the Ruby memory heap. Note that you can also use an external process monitor to kill processes with memory bloat; the process manager will restart any process regardless of how it exits.
|
|
48
48
|
|
|
49
49
|
## Usage
|
|
50
50
|
|
|
@@ -109,6 +109,18 @@ or
|
|
|
109
109
|
SIDEKIQ_MAX_MEMORY=2000m SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
|
|
110
110
|
```
|
|
111
111
|
|
|
112
|
+
Memory is checked once a minute by default. You can change how often with the `--memory-check-interval` argument or with the `SIDEKIQ_MEMORY_CHECK_INTERVAL` environment variable. The value is in seconds. Note that only one process is restarted per check, so the interval also controls how quickly multiple bloated processes are recycled.
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
bundle exec sidekiq-process-manager --processes 4 --max-memory 2g --memory-check-interval 10
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
or
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
SIDEKIQ_MEMORY_CHECK_INTERVAL=10 SIDEKIQ_MAX_MEMORY=2g SIDEKIQ_PROCESSES=4 bundle exec sidekiq-process-manager
|
|
122
|
+
```
|
|
123
|
+
|
|
112
124
|
## Alternatives
|
|
113
125
|
|
|
114
126
|
Any process manager can be an alternative (service, update, systemd, monit, god, etc.).
|
|
@@ -139,6 +151,20 @@ Or install it yourself as:
|
|
|
139
151
|
$ gem install sidekiq-process_manager
|
|
140
152
|
```
|
|
141
153
|
|
|
154
|
+
## Test Application
|
|
155
|
+
|
|
156
|
+
There is a test application in the `test_app` directory that can be used to test the process manager. You can run it with:
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
$ cd test_app
|
|
160
|
+
$ bundle install
|
|
161
|
+
$ bin/run
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
This will bring up a valkey docker instance, sidekiq cluster, and rack application that shows the status of the sidekiq processes.
|
|
165
|
+
|
|
166
|
+
Then visit [http://localhost:9292](http://localhost:9292) to see the status of the sidekiq processes.
|
|
167
|
+
|
|
142
168
|
## Contributing
|
|
143
169
|
|
|
144
170
|
Open a pull request on [GitHub](https://github.com/bdurand/sidekiq-process_manager).
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
1.1.
|
|
1
|
+
1.1.3
|
data/bin/sidekiq-process-manager
CHANGED
|
@@ -7,19 +7,23 @@ require_relative "../lib/sidekiq-process_manager"
|
|
|
7
7
|
DEFAULT_PROCESS_COUNT = 1
|
|
8
8
|
|
|
9
9
|
def parse_max_memory(max_memory)
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
10
|
+
Sidekiq::ProcessManager.parse_max_memory(max_memory)
|
|
11
|
+
rescue ArgumentError => e
|
|
12
|
+
abort(e.message)
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def parse_memory_check_interval(interval)
|
|
16
|
+
interval = interval.to_s.strip
|
|
17
|
+
return nil if interval.empty?
|
|
18
|
+
|
|
19
|
+
begin
|
|
20
|
+
interval = Float(interval)
|
|
21
|
+
rescue ArgumentError, TypeError
|
|
22
|
+
abort("Invalid memory check interval value: #{interval.inspect}")
|
|
20
23
|
end
|
|
24
|
+
abort("Memory check interval must be greater than 0") unless interval > 0
|
|
21
25
|
|
|
22
|
-
|
|
26
|
+
interval
|
|
23
27
|
end
|
|
24
28
|
|
|
25
29
|
options = {
|
|
@@ -27,6 +31,7 @@ options = {
|
|
|
27
31
|
prefork: !ENV.fetch("SIDEKIQ_PREFORK", "").empty?,
|
|
28
32
|
preboot: ENV["SIDEKIQ_PREBOOT"],
|
|
29
33
|
max_memory: parse_max_memory(ENV["SIDEKIQ_MAX_MEMORY"]),
|
|
34
|
+
memory_check_interval: parse_memory_check_interval(ENV["SIDEKIQ_MEMORY_CHECK_INTERVAL"]),
|
|
30
35
|
mode: nil,
|
|
31
36
|
}
|
|
32
37
|
|
|
@@ -45,10 +50,14 @@ parser = OptionParser.new do |opts|
|
|
|
45
50
|
options[:preboot] = preboot
|
|
46
51
|
end
|
|
47
52
|
|
|
48
|
-
opts.on('--max-memory MEMORY', "Max memory for each process (can also specify with SIDEKIQ_MAX_MEMORY); suffix with m or g to specify megabytes or gigabytes") do |max_memory|
|
|
53
|
+
opts.on('--max-memory MEMORY', "Max memory for each process (can also specify with SIDEKIQ_MAX_MEMORY); value is in bytes, or suffix with m or g to specify megabytes or gigabytes") do |max_memory|
|
|
49
54
|
options[:max_memory] = parse_max_memory(max_memory)
|
|
50
55
|
end
|
|
51
56
|
|
|
57
|
+
opts.on('--memory-check-interval SECONDS', "Seconds between memory checks when a max memory is set (can also specify with SIDEKIQ_MEMORY_CHECK_INTERVAL); defaults to #{Sidekiq::ProcessManager::Manager::DEFAULT_MEMORY_CHECK_INTERVAL} seconds") do |interval|
|
|
58
|
+
options[:memory_check_interval] = parse_memory_check_interval(interval)
|
|
59
|
+
end
|
|
60
|
+
|
|
52
61
|
opts.on('--testing', "Enable test mode") do |testing|
|
|
53
62
|
options[:mode] = :testing if testing
|
|
54
63
|
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "sidekiq"
|
|
4
4
|
require "get_process_mem"
|
|
5
|
+
require "timeout"
|
|
5
6
|
|
|
6
7
|
module Sidekiq
|
|
7
8
|
module ProcessManager
|
|
@@ -9,6 +10,12 @@ module Sidekiq
|
|
|
9
10
|
# that the specified number of sidekiq processes are running. It will also forward
|
|
10
11
|
# signals sent to the main process to the child processes.
|
|
11
12
|
class Manager
|
|
13
|
+
# Signals that are trapped by the process manager and forwarded to the child processes.
|
|
14
|
+
SIGNALS = [:INT, :TERM, :USR1, :USR2, :TSTP, :TTIN].freeze
|
|
15
|
+
|
|
16
|
+
# Number of seconds between memory checks when a max memory is set.
|
|
17
|
+
DEFAULT_MEMORY_CHECK_INTERVAL = 60
|
|
18
|
+
|
|
12
19
|
attr_reader :cli
|
|
13
20
|
|
|
14
21
|
# Create a new process manager.
|
|
@@ -16,31 +23,38 @@ module Sidekiq
|
|
|
16
23
|
# @param process_count [Integer] The number of sidekiq processes to start.
|
|
17
24
|
# @param prefork [Boolean] If true, the process manager will load the application before forking.
|
|
18
25
|
# @param preboot [String] If set, the process manager will require the specified file before forking the child processes.
|
|
26
|
+
# @param memory_check_interval [Numeric] The number of seconds between memory checks
|
|
27
|
+
# when a max memory is set.
|
|
19
28
|
# @param mode [Symbol] If set to :testing, the process manager will use a mock CLI.
|
|
20
29
|
# @param silent [Boolean] If true, the process manager will not output any messages.
|
|
21
|
-
def initialize(process_count: 1, prefork: false, preboot: nil, max_memory: nil, mode: nil, silent: false)
|
|
30
|
+
def initialize(process_count: 1, prefork: false, preboot: nil, max_memory: nil, memory_check_interval: nil, mode: nil, silent: false)
|
|
22
31
|
require "sidekiq/cli"
|
|
23
32
|
|
|
24
33
|
# Get the number of processes to fork
|
|
25
34
|
@process_count = process_count
|
|
26
|
-
raise ArgumentError.new("Process count must be greater than
|
|
35
|
+
raise ArgumentError.new("Process count must be greater than 0") if @process_count < 1
|
|
27
36
|
|
|
28
|
-
@prefork =
|
|
37
|
+
@prefork = prefork && process_count > 1
|
|
29
38
|
@preboot = preboot if process_count > 1 && !prefork
|
|
30
39
|
@max_memory = ((max_memory.to_i > 0) ? max_memory.to_i : nil)
|
|
31
40
|
|
|
41
|
+
if !memory_check_interval.nil? && memory_check_interval.to_f <= 0
|
|
42
|
+
raise ArgumentError.new("Memory check interval must be greater than 0")
|
|
43
|
+
end
|
|
44
|
+
|
|
32
45
|
if mode == :testing
|
|
33
46
|
require_relative "../../../spec/support/mocks"
|
|
34
47
|
@cli = MockSidekiqCLI.new(silent)
|
|
35
|
-
@memory_check_interval = 1
|
|
48
|
+
@memory_check_interval = (memory_check_interval || 1).to_f
|
|
49
|
+
@exit_grace_period = 0.5
|
|
36
50
|
else
|
|
37
51
|
@cli = Sidekiq::CLI.instance
|
|
38
|
-
@memory_check_interval =
|
|
52
|
+
@memory_check_interval = (memory_check_interval || DEFAULT_MEMORY_CHECK_INTERVAL).to_f
|
|
53
|
+
@exit_grace_period = 5
|
|
39
54
|
end
|
|
40
55
|
|
|
41
56
|
@silent = silent
|
|
42
57
|
@pids = []
|
|
43
|
-
@terminated_pids = []
|
|
44
58
|
@started = false
|
|
45
59
|
@mutex = Mutex.new
|
|
46
60
|
end
|
|
@@ -69,18 +83,22 @@ module Sidekiq
|
|
|
69
83
|
|
|
70
84
|
while @signal_pipe_read.wait_readable
|
|
71
85
|
begin
|
|
72
|
-
signal = @signal_pipe_read.gets
|
|
86
|
+
signal = @signal_pipe_read.gets
|
|
87
|
+
break if signal.nil?
|
|
88
|
+
|
|
89
|
+
signal = signal.strip
|
|
73
90
|
send_signal_to_children(signal.to_sym)
|
|
74
91
|
rescue => e
|
|
75
|
-
|
|
92
|
+
log_warning("Error sending signal #{signal} to child processes: #{e.message}")
|
|
76
93
|
end
|
|
77
94
|
end
|
|
78
95
|
end
|
|
79
96
|
|
|
80
97
|
# Trap signals that will be forwarded to child processes
|
|
81
|
-
|
|
98
|
+
SIGNALS.each do |signal|
|
|
82
99
|
::Signal.trap(signal) do
|
|
83
100
|
if ::Process.pid == master_pid
|
|
101
|
+
signal = :TERM if signal == :INT
|
|
84
102
|
@signal_pipe_write.puts(signal)
|
|
85
103
|
end
|
|
86
104
|
end
|
|
@@ -88,15 +106,19 @@ module Sidekiq
|
|
|
88
106
|
|
|
89
107
|
# Ensure that child processes receive the term signal when the master process exits.
|
|
90
108
|
at_exit do
|
|
91
|
-
if ::Process.pid == master_pid
|
|
92
|
-
|
|
109
|
+
if ::Process.pid == master_pid
|
|
110
|
+
if @process_count > 0
|
|
111
|
+
send_signal_to_children(:TERM)
|
|
112
|
+
end
|
|
113
|
+
wait_for_children_to_exit
|
|
114
|
+
log_info("Process manager exiting")
|
|
93
115
|
end
|
|
94
116
|
end
|
|
95
117
|
|
|
96
118
|
GC.start
|
|
97
119
|
GC.compact if GC.respond_to?(:compact)
|
|
98
120
|
# I'm not sure why, but running GC operations blocks until we try to write some I/O.
|
|
99
|
-
File.write(
|
|
121
|
+
File.write(File::NULL, "0")
|
|
100
122
|
|
|
101
123
|
@process_count.times do
|
|
102
124
|
start_child_process!
|
|
@@ -104,9 +126,8 @@ module Sidekiq
|
|
|
104
126
|
|
|
105
127
|
start_memory_monitor if @max_memory
|
|
106
128
|
|
|
107
|
-
log_info("Process manager started
|
|
129
|
+
log_info("Process manager started")
|
|
108
130
|
monitor_child_processes
|
|
109
|
-
log_info("Process manager #{::Process.pid} exiting")
|
|
110
131
|
end
|
|
111
132
|
|
|
112
133
|
# Helper to wait on the manager to wait on child processes to start up.
|
|
@@ -116,7 +137,7 @@ module Sidekiq
|
|
|
116
137
|
def wait(timeout = 5)
|
|
117
138
|
timeout_time = monotonic_time + timeout
|
|
118
139
|
while monotonic_time <= timeout_time
|
|
119
|
-
return if
|
|
140
|
+
return if pids.size == @process_count
|
|
120
141
|
sleep(0.01)
|
|
121
142
|
end
|
|
122
143
|
|
|
@@ -213,11 +234,20 @@ module Sidekiq
|
|
|
213
234
|
end
|
|
214
235
|
|
|
215
236
|
def set_program_name!
|
|
216
|
-
$PROGRAM_NAME = "sidekiq process manager #{sidekiq_options[:tag]} [#{
|
|
237
|
+
$PROGRAM_NAME = "sidekiq process manager #{sidekiq_options[:tag]} [#{pids.size} processes]"
|
|
217
238
|
end
|
|
218
239
|
|
|
219
240
|
def start_child_process!
|
|
220
241
|
pid = fork do
|
|
242
|
+
# Child processes inherit the signal handlers installed by the process
|
|
243
|
+
# manager. Those handlers only do anything in the master process, so
|
|
244
|
+
# leaving them in place would silently discard any signal sent to the
|
|
245
|
+
# child before sidekiq installs its own handlers. Restore the default
|
|
246
|
+
# behavior until then so signals are never swallowed.
|
|
247
|
+
SIGNALS.each do |signal|
|
|
248
|
+
::Signal.trap(signal, "DEFAULT")
|
|
249
|
+
end
|
|
250
|
+
|
|
221
251
|
# Set $0 so instrumentation libraries detecting sidekiq from the command run will work properly.
|
|
222
252
|
$0 = File.join(File.dirname($0), "sidekiq")
|
|
223
253
|
@process_count = 0
|
|
@@ -228,27 +258,47 @@ module Sidekiq
|
|
|
228
258
|
Sidekiq::ProcessManager.run_after_fork_hooks
|
|
229
259
|
@cli.run
|
|
230
260
|
end
|
|
231
|
-
|
|
261
|
+
shutting_down = false
|
|
262
|
+
@mutex.synchronize do
|
|
263
|
+
@pids << pid
|
|
264
|
+
shutting_down = @process_count < 1
|
|
265
|
+
end
|
|
232
266
|
log_info("Forked sidekiq process with pid #{pid}")
|
|
267
|
+
# A shutdown signal may have been forwarded to the other processes while
|
|
268
|
+
# this one was being forked; it needs to get the signal as well.
|
|
269
|
+
send_signal_to_pid(:TERM, pid) if shutting_down
|
|
233
270
|
set_program_name!
|
|
234
271
|
end
|
|
235
272
|
|
|
236
273
|
def send_signal_to_children(signal)
|
|
237
274
|
log_info("Process manager trapped signal #{signal}")
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
275
|
+
child_pids = @mutex.synchronize do
|
|
276
|
+
@process_count = 0 if signal == :INT || signal == :TERM
|
|
277
|
+
@pids.dup
|
|
278
|
+
end
|
|
279
|
+
child_pids.each do |pid|
|
|
280
|
+
send_signal_to_pid(signal, pid)
|
|
281
|
+
end
|
|
282
|
+
end
|
|
283
|
+
|
|
284
|
+
def send_signal_to_pid(signal, pid)
|
|
285
|
+
signal = signal.to_sym
|
|
286
|
+
begin
|
|
287
|
+
log_info("Sending signal #{signal} to sidekiq process #{pid}")
|
|
288
|
+
::Process.kill(signal, pid)
|
|
289
|
+
if [:TERM, :INT].include?(signal)
|
|
290
|
+
Thread.new do
|
|
291
|
+
Thread.current.name = "pid-#{pid}-killer"
|
|
292
|
+
ensure_pid_dies(pid)
|
|
293
|
+
end
|
|
245
294
|
end
|
|
295
|
+
rescue Errno::ESRCH
|
|
296
|
+
# The process is already dead
|
|
246
297
|
end
|
|
247
|
-
wait_for_children_to_exit(pids) if @process_count == 0
|
|
248
298
|
end
|
|
249
299
|
|
|
250
300
|
def start_memory_monitor
|
|
251
|
-
log_info("Starting memory monitor with max memory #{(@max_memory / (1024**2)).round}mb")
|
|
301
|
+
log_info("Starting memory monitor with max memory #{(@max_memory / (1024**2)).round}mb checked every #{@memory_check_interval} seconds")
|
|
252
302
|
|
|
253
303
|
@memory_monitor = Thread.new do
|
|
254
304
|
Thread.current.name = "memory-monitor"
|
|
@@ -256,20 +306,14 @@ module Sidekiq
|
|
|
256
306
|
sleep(@memory_check_interval)
|
|
257
307
|
|
|
258
308
|
pids.each do |pid|
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
::Process.kill(:TERM, pid)
|
|
265
|
-
rescue Errno::ESRCH
|
|
266
|
-
# The process is already dead
|
|
267
|
-
end
|
|
268
|
-
break
|
|
269
|
-
end
|
|
270
|
-
rescue => e
|
|
271
|
-
log_warning("Error monitoring memory for sidekiq process #{pid}: #{e.inspect}")
|
|
309
|
+
memory = GetProcessMem.new(pid)
|
|
310
|
+
if memory.bytes > @max_memory
|
|
311
|
+
log_warning("Killing bloated sidekiq process #{pid}: #{memory.mb.round}mb used")
|
|
312
|
+
send_signal_to_pid(:TERM, pid)
|
|
313
|
+
break
|
|
272
314
|
end
|
|
315
|
+
rescue => e
|
|
316
|
+
log_warning("Error monitoring memory for sidekiq process #{pid}: #{e.inspect}")
|
|
273
317
|
end
|
|
274
318
|
end
|
|
275
319
|
end
|
|
@@ -281,49 +325,122 @@ module Sidekiq
|
|
|
281
325
|
end
|
|
282
326
|
end
|
|
283
327
|
|
|
284
|
-
def wait_for_children_to_exit
|
|
285
|
-
|
|
286
|
-
pids
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
328
|
+
def wait_for_children_to_exit
|
|
329
|
+
# Allow the SIGKILL failsafe in ensure_pid_dies time to fire before giving up on a process.
|
|
330
|
+
remaining = reap_child_processes(pids, monotonic_time + sidekiq_shutdown_timeout + @exit_grace_period)
|
|
331
|
+
|
|
332
|
+
unless remaining.empty?
|
|
333
|
+
# Don't leave orphaned processes behind. These pids were never reaped,
|
|
334
|
+
# so the operating system cannot have reused them for another process
|
|
335
|
+
# and it is safe to signal them.
|
|
336
|
+
remaining.each do |pid|
|
|
337
|
+
::Process.kill(:KILL, pid)
|
|
338
|
+
log_warning("Sidekiq process #{pid} failed to exit; killed with SIGKILL")
|
|
339
|
+
rescue Errno::ESRCH
|
|
340
|
+
# The process is already dead
|
|
290
341
|
end
|
|
342
|
+
|
|
343
|
+
# SIGKILL cannot be caught or ignored, so the processes will exit, but
|
|
344
|
+
# not necessarily by the time the signal is sent. Wait for them so they
|
|
345
|
+
# are not left behind as zombies.
|
|
346
|
+
remaining = reap_child_processes(remaining, monotonic_time + @exit_grace_period)
|
|
291
347
|
end
|
|
292
348
|
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
# Ignore errors so we can continue to kill other processes.
|
|
298
|
-
end
|
|
349
|
+
if remaining.empty?
|
|
350
|
+
log_info("All sidekiq processes have exited")
|
|
351
|
+
else
|
|
352
|
+
log_warning("Gave up waiting on sidekiq processes: #{remaining.join(", ")}")
|
|
299
353
|
end
|
|
300
354
|
end
|
|
301
355
|
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
356
|
+
# Reap child processes as they exit until they have all been reaped or the
|
|
357
|
+
# deadline passes.
|
|
358
|
+
#
|
|
359
|
+
# @return [Array<Integer>] The pids that were not reaped.
|
|
360
|
+
def reap_child_processes(child_pids, deadline)
|
|
361
|
+
remaining = child_pids.dup
|
|
362
|
+
until remaining.empty?
|
|
363
|
+
# Poll all of the processes on each pass so that one slow process
|
|
364
|
+
# cannot consume the entire timeout waiting for the others.
|
|
365
|
+
remaining.delete_if { |pid| reap_child_process(pid) }
|
|
366
|
+
break if remaining.empty? || monotonic_time >= deadline
|
|
367
|
+
sleep(0.01)
|
|
368
|
+
end
|
|
369
|
+
remaining
|
|
370
|
+
end
|
|
371
|
+
|
|
372
|
+
# Reap a child process if it has exited. The pid is only removed from the
|
|
373
|
+
# list of running processes when it is actually reaped so that the SIGKILL
|
|
374
|
+
# failsafe in ensure_pid_dies still sees it as running until then.
|
|
375
|
+
#
|
|
376
|
+
# @return [Boolean] true if the process has been reaped or is no longer a child process.
|
|
377
|
+
def reap_child_process(pid)
|
|
378
|
+
reaped = begin
|
|
379
|
+
# The monitoring loop may no longer be running, so reap the child
|
|
380
|
+
# process here rather than just polling for it to disappear.
|
|
381
|
+
::Process.waitpid(pid, ::Process::WNOHANG)
|
|
382
|
+
rescue Errno::ECHILD, Errno::ESRCH
|
|
383
|
+
# The process has already been reaped.
|
|
384
|
+
pid
|
|
385
|
+
end
|
|
386
|
+
return false unless reaped
|
|
387
|
+
|
|
388
|
+
@mutex.synchronize { @pids.delete(pid) }
|
|
389
|
+
true
|
|
390
|
+
end
|
|
391
|
+
|
|
392
|
+
def sidekiq_shutdown_timeout
|
|
393
|
+
(sidekiq_options[:timeout] || 25).to_f
|
|
394
|
+
end
|
|
395
|
+
|
|
396
|
+
def ensure_pid_dies(pid)
|
|
397
|
+
# Wait for the process to die, or kill it after a timeout. A pid is
|
|
398
|
+
# removed from the list only once it has been reaped, so the list is the
|
|
399
|
+
# source of truth for liveness; a pid still in it cannot have been reused
|
|
400
|
+
# by the operating system for a different process. Checking with a system
|
|
401
|
+
# call is not reliable since it cannot distinguish a recycled pid.
|
|
402
|
+
end_time = monotonic_time + sidekiq_shutdown_timeout
|
|
403
|
+
while monotonic_time < end_time && pids.include?(pid)
|
|
404
|
+
sleep(0.1)
|
|
405
|
+
end
|
|
406
|
+
|
|
407
|
+
if pids.include?(pid)
|
|
408
|
+
begin
|
|
409
|
+
::Process.kill(:KILL, pid)
|
|
410
|
+
log_warning("Sidekiq process #{pid} failed to exit after #{sidekiq_shutdown_timeout} seconds; killed with SIGKILL")
|
|
411
|
+
rescue Errno::ESRCH
|
|
412
|
+
# The process is already dead
|
|
413
|
+
end
|
|
308
414
|
end
|
|
309
415
|
end
|
|
310
416
|
|
|
311
417
|
# Listen for child processes dying and restart if necessary.
|
|
312
418
|
def monitor_child_processes
|
|
313
419
|
loop do
|
|
314
|
-
|
|
315
|
-
|
|
420
|
+
begin
|
|
421
|
+
pid = ::Process.wait
|
|
422
|
+
rescue Errno::ECHILD
|
|
423
|
+
# All child processes are already gone.
|
|
424
|
+
@mutex.synchronize { @pids.clear }
|
|
425
|
+
break
|
|
426
|
+
end
|
|
427
|
+
|
|
428
|
+
# Process.wait reaps any child process, including ones spawned by
|
|
429
|
+
# libraries or application code (i.e. get_process_mem shelling out
|
|
430
|
+
# to ps). Only act on processes forked by the manager.
|
|
431
|
+
removed = @mutex.synchronize { @pids.delete(pid) }
|
|
432
|
+
next unless removed
|
|
316
433
|
|
|
317
434
|
log_info("Sidekiq process #{pid} exited")
|
|
318
435
|
|
|
319
436
|
# If there are not enough processes running, start a replacement one.
|
|
320
|
-
if @process_count >
|
|
437
|
+
if @process_count > pids.size
|
|
321
438
|
start_child_process!
|
|
322
439
|
end
|
|
323
440
|
|
|
324
441
|
set_program_name!
|
|
325
442
|
|
|
326
|
-
if
|
|
443
|
+
if pids.empty?
|
|
327
444
|
break
|
|
328
445
|
end
|
|
329
446
|
end
|
|
@@ -33,6 +33,31 @@ module Sidekiq
|
|
|
33
33
|
end
|
|
34
34
|
@after_fork = nil
|
|
35
35
|
end
|
|
36
|
+
|
|
37
|
+
# Parse a max memory value into a number of bytes. The value can be a plain
|
|
38
|
+
# number of bytes or a number suffixed with "m" for megabytes or "g" for
|
|
39
|
+
# gigabytes (case insensitive).
|
|
40
|
+
#
|
|
41
|
+
# @param value [String, nil] The value to parse.
|
|
42
|
+
# @return [Float, nil] The number of bytes or nil if the value is blank.
|
|
43
|
+
# @raise [ArgumentError] If the value cannot be parsed.
|
|
44
|
+
def parse_max_memory(value)
|
|
45
|
+
value = value.to_s.strip
|
|
46
|
+
return nil if value.empty?
|
|
47
|
+
|
|
48
|
+
matched = value.match(/\A(\d+(?:\.\d+)?)([mg]?)\z/i)
|
|
49
|
+
raise ArgumentError.new("Invalid max memory value: #{value.inspect}") unless matched
|
|
50
|
+
|
|
51
|
+
bytes = matched[1].to_f
|
|
52
|
+
case matched[2].downcase
|
|
53
|
+
when "m"
|
|
54
|
+
bytes *= 1024**2
|
|
55
|
+
when "g"
|
|
56
|
+
bytes *= 1024**3
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
bytes
|
|
60
|
+
end
|
|
36
61
|
end
|
|
37
62
|
end
|
|
38
63
|
end
|
|
@@ -7,9 +7,16 @@ Gem::Specification.new do |spec|
|
|
|
7
7
|
spec.email = ["bbdurand@gmail.com"]
|
|
8
8
|
|
|
9
9
|
spec.summary = "Process manager for forking and monitoring multiple sidekiq processes."
|
|
10
|
+
|
|
10
11
|
spec.homepage = "https://github.com/bdurand/sidekiq-process_manager"
|
|
11
12
|
spec.license = "MIT"
|
|
12
13
|
|
|
14
|
+
spec.metadata = {
|
|
15
|
+
"homepage_uri" => spec.homepage,
|
|
16
|
+
"source_code_uri" => spec.homepage,
|
|
17
|
+
"changelog_uri" => "#{spec.homepage}/blob/main/CHANGELOG.md"
|
|
18
|
+
}
|
|
19
|
+
|
|
13
20
|
# Specify which files should be added to the gem when it is released.
|
|
14
21
|
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
|
15
22
|
ignore_files = %w[
|
|
@@ -20,6 +27,7 @@ Gem::Specification.new do |spec|
|
|
|
20
27
|
Rakefile
|
|
21
28
|
gemfiles/
|
|
22
29
|
spec/
|
|
30
|
+
test_app/
|
|
23
31
|
]
|
|
24
32
|
spec.files = Dir.chdir(File.expand_path("..", __FILE__)) do
|
|
25
33
|
`git ls-files -z`.split("\x0").reject { |f| ignore_files.any? { |path| f.start_with?(path) } }
|
|
@@ -33,6 +41,4 @@ Gem::Specification.new do |spec|
|
|
|
33
41
|
|
|
34
42
|
spec.add_dependency "sidekiq", ">= 5.0"
|
|
35
43
|
spec.add_dependency "get_process_mem"
|
|
36
|
-
|
|
37
|
-
spec.add_development_dependency "bundler"
|
|
38
44
|
end
|
metadata
CHANGED
|
@@ -1,14 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: sidekiq-process_manager
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.1.
|
|
4
|
+
version: 1.1.3
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Brian Durand
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
13
|
name: sidekiq
|
|
@@ -38,21 +37,6 @@ dependencies:
|
|
|
38
37
|
- - ">="
|
|
39
38
|
- !ruby/object:Gem::Version
|
|
40
39
|
version: '0'
|
|
41
|
-
- !ruby/object:Gem::Dependency
|
|
42
|
-
name: bundler
|
|
43
|
-
requirement: !ruby/object:Gem::Requirement
|
|
44
|
-
requirements:
|
|
45
|
-
- - ">="
|
|
46
|
-
- !ruby/object:Gem::Version
|
|
47
|
-
version: '0'
|
|
48
|
-
type: :development
|
|
49
|
-
prerelease: false
|
|
50
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
51
|
-
requirements:
|
|
52
|
-
- - ">="
|
|
53
|
-
- !ruby/object:Gem::Version
|
|
54
|
-
version: '0'
|
|
55
|
-
description:
|
|
56
40
|
email:
|
|
57
41
|
- bbdurand@gmail.com
|
|
58
42
|
executables:
|
|
@@ -73,8 +57,10 @@ files:
|
|
|
73
57
|
homepage: https://github.com/bdurand/sidekiq-process_manager
|
|
74
58
|
licenses:
|
|
75
59
|
- MIT
|
|
76
|
-
metadata:
|
|
77
|
-
|
|
60
|
+
metadata:
|
|
61
|
+
homepage_uri: https://github.com/bdurand/sidekiq-process_manager
|
|
62
|
+
source_code_uri: https://github.com/bdurand/sidekiq-process_manager
|
|
63
|
+
changelog_uri: https://github.com/bdurand/sidekiq-process_manager/blob/main/CHANGELOG.md
|
|
78
64
|
rdoc_options: []
|
|
79
65
|
require_paths:
|
|
80
66
|
- lib
|
|
@@ -89,8 +75,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
89
75
|
- !ruby/object:Gem::Version
|
|
90
76
|
version: '0'
|
|
91
77
|
requirements: []
|
|
92
|
-
rubygems_version:
|
|
93
|
-
signing_key:
|
|
78
|
+
rubygems_version: 4.0.3
|
|
94
79
|
specification_version: 4
|
|
95
80
|
summary: Process manager for forking and monitoring multiple sidekiq processes.
|
|
96
81
|
test_files: []
|