pitchfork 0.9.0 → 0.10.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +4 -0
- data/Gemfile.lock +1 -1
- data/docs/REFORKING.md +13 -4
- data/lib/pitchfork/children.rb +27 -0
- data/lib/pitchfork/http_server.rb +25 -30
- data/lib/pitchfork/version.rb +1 -1
- data/lib/pitchfork/worker.rb +8 -0
- data/lib/pitchfork.rb +18 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 0f23c677bd64eb8ca7a040cb2d41dab4047e05ae849da16af8e537748bfb6dbe
|
4
|
+
data.tar.gz: 890bac5b67c23d6f0aacf0d9130999cf69ac0bc5a2f1ea06bdd142d67c0606ac
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 39c33bbbe865ba22bebc8b6b1e823f71d0ba6b2787f06af26f0ffe3d4251c781850826fcbcdf1dc13b1734919770e35dc57815d2de057f18d97ba354f247a3e7
|
7
|
+
data.tar.gz: 94e5ae39e129af1fd950c9a77e094bac5bda586f7358115fd8cf25bb3c6ee2079bb4654ebd4e6ed5e9ff498b461559da07d7b44acc0147fcac5c9a4695b7c62c
|
data/CHANGELOG.md
CHANGED
data/Gemfile.lock
CHANGED
data/docs/REFORKING.md
CHANGED
@@ -66,7 +66,10 @@ PID COMMAND
|
|
66
66
|
105 \_ pitchfork (gen:0) worker[3]
|
67
67
|
```
|
68
68
|
|
69
|
-
|
69
|
+
As the diagram shows, while workers are forked from the mold, they become children of the master process.
|
70
|
+
We'll see how does that work [later](#forking-sibling-processes).
|
71
|
+
|
72
|
+
When a reforking is triggered, one of the workers is selected to fork a new `mold`:
|
70
73
|
|
71
74
|
```
|
72
75
|
PID COMMAND
|
@@ -79,6 +82,9 @@ PID COMMAND
|
|
79
82
|
105 \_ pitchfork (gen:1) mold
|
80
83
|
```
|
81
84
|
|
85
|
+
Again, while the mold was forked from a worker, it becomes a child of the master process.
|
86
|
+
We'll see how does that work [later](#forking-sibling-processes).
|
87
|
+
|
82
88
|
When that new mold is ready, `pitchfork` terminates the old mold and starts a slow rollout of older workers and replace them with fresh workers
|
83
89
|
forked from the mold:
|
84
90
|
|
@@ -104,7 +110,7 @@ PID COMMAND
|
|
104
110
|
|
105
111
|
etc.
|
106
112
|
|
107
|
-
### Forking Sibling Processes
|
113
|
+
### Forking Sibling Processes
|
108
114
|
|
109
115
|
Normally on unix systems, when calling `fork(2)`, the newly created process is a child of the original one, so forking from the mold should create
|
110
116
|
a process tree such as:
|
@@ -119,5 +125,8 @@ PID COMMAND
|
|
119
125
|
However the `pitchfork` master process registers itself as a "child subreaper" via [`PR_SET_CHILD_SUBREAPER`](https://man7.org/linux/man-pages/man2/prctl.2.html).
|
120
126
|
This means any descendant process that is orphaned will be re-parented as a child of the master rather than a child of the init process (pid 1).
|
121
127
|
|
122
|
-
With this in mind, the mold
|
123
|
-
|
128
|
+
With this in mind, the mold forks twice to create an orphaned process that will get re-attached to the master,
|
129
|
+
effectively forking a sibling rather than a child. Similarly, workers do the same when forking new molds.
|
130
|
+
This technique eases killing previous generations of molds and workers.
|
131
|
+
|
132
|
+
The need for `PR_SET_CHILD_SUBREAPER` is the main reason why reforking is only available on Linux.
|
data/lib/pitchfork/children.rb
CHANGED
@@ -107,6 +107,33 @@ module Pitchfork
|
|
107
107
|
@workers.each_value(&block)
|
108
108
|
end
|
109
109
|
|
110
|
+
def soft_kill_all(sig)
|
111
|
+
each do |child|
|
112
|
+
child.soft_kill(sig)
|
113
|
+
end
|
114
|
+
end
|
115
|
+
|
116
|
+
def hard_kill(sig, child)
|
117
|
+
child.hard_kill(sig)
|
118
|
+
rescue Errno::ESRCH
|
119
|
+
reap(child.pid)
|
120
|
+
child.close
|
121
|
+
end
|
122
|
+
|
123
|
+
def hard_kill_all(sig)
|
124
|
+
each do |child|
|
125
|
+
hard_kill(sig, child)
|
126
|
+
end
|
127
|
+
end
|
128
|
+
|
129
|
+
def hard_timeout(child)
|
130
|
+
child.hard_timeout!
|
131
|
+
rescue Errno::ESRCH
|
132
|
+
reap(child.pid)
|
133
|
+
child.close
|
134
|
+
true
|
135
|
+
end
|
136
|
+
|
110
137
|
def workers
|
111
138
|
@workers.values
|
112
139
|
end
|
@@ -396,15 +396,15 @@ module Pitchfork
|
|
396
396
|
limit = Pitchfork.time_now + timeout
|
397
397
|
until @children.workers.empty? || Pitchfork.time_now > limit
|
398
398
|
if graceful
|
399
|
-
|
399
|
+
@children.soft_kill_all(:TERM)
|
400
400
|
else
|
401
|
-
|
401
|
+
@children.hard_kill_all(:INT)
|
402
402
|
end
|
403
403
|
if monitor_loop(false) == StopIteration
|
404
404
|
return StopIteration
|
405
405
|
end
|
406
406
|
end
|
407
|
-
|
407
|
+
@children.hard_kill_all(:KILL)
|
408
408
|
@promotion_lock.unlink
|
409
409
|
end
|
410
410
|
|
@@ -506,25 +506,28 @@ module Pitchfork
|
|
506
506
|
next
|
507
507
|
else # worker is out of time
|
508
508
|
next_sleep = 0
|
509
|
-
|
510
|
-
|
511
|
-
|
512
|
-
logger.error "worker=#{worker.nr} pid=#{worker.pid} timed out, killing"
|
513
|
-
end
|
509
|
+
hard_timeout(worker)
|
510
|
+
end
|
511
|
+
end
|
514
512
|
|
515
|
-
|
516
|
-
|
517
|
-
@after_worker_hard_timeout.call(self, worker)
|
518
|
-
rescue => error
|
519
|
-
Pitchfork.log_error(@logger, "after_worker_hard_timeout callback", error)
|
520
|
-
end
|
521
|
-
end
|
513
|
+
next_sleep <= 0 ? 1 : next_sleep
|
514
|
+
end
|
522
515
|
|
523
|
-
|
516
|
+
def hard_timeout(worker)
|
517
|
+
if @after_worker_hard_timeout
|
518
|
+
begin
|
519
|
+
@after_worker_hard_timeout.call(self, worker)
|
520
|
+
rescue => error
|
521
|
+
Pitchfork.log_error(@logger, "after_worker_hard_timeout callback", error)
|
524
522
|
end
|
525
523
|
end
|
526
524
|
|
527
|
-
|
525
|
+
if worker.mold?
|
526
|
+
logger.error "mold pid=#{worker.pid} timed out, killing"
|
527
|
+
else
|
528
|
+
logger.error "worker=#{worker.nr} pid=#{worker.pid} timed out, killing"
|
529
|
+
end
|
530
|
+
@children.hard_timeout(worker) # take no prisoners for hard timeout violations
|
528
531
|
end
|
529
532
|
|
530
533
|
def trigger_refork
|
@@ -697,12 +700,12 @@ module Pitchfork
|
|
697
700
|
|
698
701
|
# once a client is accepted, it is processed in its entirety here
|
699
702
|
# in 3 easy steps: read request, call app, write app response
|
700
|
-
def process_client(client, timeout_handler)
|
703
|
+
def process_client(client, worker, timeout_handler)
|
701
704
|
env = nil
|
702
705
|
@request = Pitchfork::HttpParser.new
|
703
706
|
env = @request.read(client)
|
704
707
|
|
705
|
-
proc_name status: "processing: #{env["PATH_INFO"]}"
|
708
|
+
proc_name status: "requests: #{worker.requests_count}, processing: #{env["PATH_INFO"]}"
|
706
709
|
|
707
710
|
timeout_handler.rack_env = env
|
708
711
|
env["pitchfork.timeout"] = timeout_handler
|
@@ -835,7 +838,7 @@ module Pitchfork
|
|
835
838
|
when Message
|
836
839
|
worker.update(client)
|
837
840
|
else
|
838
|
-
request_env = process_client(client, prepare_timeout(worker))
|
841
|
+
request_env = process_client(client, worker, prepare_timeout(worker))
|
839
842
|
@after_request_complete&.call(self, worker, request_env)
|
840
843
|
worker.increment_requests_count
|
841
844
|
end
|
@@ -848,6 +851,7 @@ module Pitchfork
|
|
848
851
|
|
849
852
|
if @refork_condition && Info.fork_safe? && !worker.outdated?
|
850
853
|
if @refork_condition.met?(worker, logger)
|
854
|
+
proc_name status: "requests: #{worker.requests_count}, spawning mold"
|
851
855
|
if spawn_mold(worker.generation)
|
852
856
|
logger.info("Refork condition met, promoting ourselves")
|
853
857
|
end
|
@@ -855,7 +859,7 @@ module Pitchfork
|
|
855
859
|
end
|
856
860
|
end
|
857
861
|
|
858
|
-
proc_name status: "waiting"
|
862
|
+
proc_name status: "requests: #{worker.requests_count}, waiting"
|
859
863
|
waiter.get_readers(ready, readers, @timeout * 500) # to milliseconds, but halved
|
860
864
|
rescue => e
|
861
865
|
Pitchfork.log_error(@logger, "listen loop error", e) if readers[0]
|
@@ -930,15 +934,6 @@ module Pitchfork
|
|
930
934
|
worker = @children.reap(wpid) and worker.close rescue nil
|
931
935
|
end
|
932
936
|
|
933
|
-
# delivers a signal to each worker
|
934
|
-
def kill_each_child(signal)
|
935
|
-
@children.each { |w| kill_worker(signal, w.pid) }
|
936
|
-
end
|
937
|
-
|
938
|
-
def soft_kill_each_child(signal)
|
939
|
-
@children.each { |worker| worker.soft_kill(signal) }
|
940
|
-
end
|
941
|
-
|
942
937
|
# returns an array of string names for the given listener array
|
943
938
|
def listener_names(listeners = LISTENERS)
|
944
939
|
listeners.map { |io| sock_name(io) }
|
data/lib/pitchfork/version.rb
CHANGED
data/lib/pitchfork/worker.rb
CHANGED
@@ -139,6 +139,14 @@ module Pitchfork
|
|
139
139
|
success
|
140
140
|
end
|
141
141
|
|
142
|
+
def hard_kill(sig)
|
143
|
+
Process.kill(sig, pid)
|
144
|
+
end
|
145
|
+
|
146
|
+
def hard_timeout!
|
147
|
+
hard_kill(:KILL)
|
148
|
+
end
|
149
|
+
|
142
150
|
# this only runs when the Rack app.call is not running
|
143
151
|
# act like a listener
|
144
152
|
def accept_nonblock(exception: nil) # :nodoc:
|
data/lib/pitchfork.rb
CHANGED
@@ -204,8 +204,13 @@ module Pitchfork
|
|
204
204
|
# or the master to be PID 1.
|
205
205
|
if middle_pid = FORK_LOCK.synchronize { Process.fork } # parent
|
206
206
|
# We need to wait(2) so that the middle process doesn't end up a zombie.
|
207
|
-
|
207
|
+
# The process only call fork again an exit so it should be pretty fast.
|
208
|
+
# However it might need to execute some `Process._fork` or `at_exit` callbacks,
|
209
|
+
# so it case it takes more than 5 seconds to exit, we kill it with SIGBUS
|
210
|
+
# to produce a crash report, as this is indicative of a nasty bug.
|
211
|
+
process_wait_with_timeout(middle_pid, 5, :BUS)
|
208
212
|
else # first child
|
213
|
+
Process.setproctitle("<pitchfork fork_sibling>")
|
209
214
|
clean_fork(&block) # detach into a grand child
|
210
215
|
exit
|
211
216
|
end
|
@@ -216,6 +221,18 @@ module Pitchfork
|
|
216
221
|
nil # it's tricky to return the PID
|
217
222
|
end
|
218
223
|
|
224
|
+
def process_wait_with_timeout(pid, timeout, timeout_signal = :KILL)
|
225
|
+
(timeout * 200).times do
|
226
|
+
status = Process.wait(pid, Process::WNOHANG)
|
227
|
+
return status if status
|
228
|
+
sleep 0.005 # 200 * 5ms => 1s
|
229
|
+
end
|
230
|
+
|
231
|
+
# The process didn't exit in the allotted time, so we kill it.
|
232
|
+
Process.kill(timeout_signal, pid)
|
233
|
+
Process.wait(pid)
|
234
|
+
end
|
235
|
+
|
219
236
|
def time_now(int = false)
|
220
237
|
Process.clock_gettime(Process::CLOCK_MONOTONIC, int ? :second : :float_second)
|
221
238
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pitchfork
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.10.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jean Boussier
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-11-02 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: raindrops
|