pitchfork 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f9f15127612a30958ee43b9667990a0728b5fdfc46e13285bdfa1706085eb67d
-  data.tar.gz: '08ba863ccf209644093b83ee905305615cfef046c4cd5e11e74f16540c722ad7'
+  metadata.gz: 3e42e6177d834ad380a59378ccea33b4110b443975d913ca6bc66c84ee5abbdb
+  data.tar.gz: 79ae3e1c674b3fa6b6bd78103d177d6593e8ce9239f4603450f300ad5685a827
 SHA512:
-  metadata.gz: 6c73e64f686acde16c5bb50df83a3a8596d9a3f578318cdc55bc3d8dbee1a0d7dd8534e627b5baa4994f3f30d4917ffe7d4e5d405850d068f7523cf52c21113b
-  data.tar.gz: 80c872414fe8289c7c98cb9949309f76aeec63b3f3de85f09349839f873721cdb41c12bfa4620c7863d62526533c7e4ee900b7f5461917d348fa79d4c5977e4a
+  metadata.gz: 1ff1761d7107bf62e2e7f31c7b1843ffe78eb99d650a8175ef0f9173f871499afb69b13407e9c70d80be4afd799a2c18243942318b92d5bd339bbbf591abb32b
+  data.tar.gz: 7c49a7909dfd4d3405f66859a308700dd047aa360412a752831348a000ecd4be58d493856539e46b56825e74a7fd19ae81731344bef0a1cde16e3529134c94eb

data/.gitignore

@@ -3,6 +3,7 @@
 *.log
 *.so
 *.rbc
+*.log
 .DS_Store
 /.config
 /InstalledFiles

data/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Unreleased
 
+# 0.3.0
+
+- Renamed `after_promotion` in `after_mold_fork`.
+- Renamed `after_fork` in `after_worker_fork`.
+- Backoff 10s after every mold spawning attempt.
+- Spawn mold from workers instead of promoting workers (#42).
+
+# 0.2.0
+
 - Remove default middlewares.
 - Refork indefinitely when `refork_after` is set, unless the last element is `false`.
 - Remove `mold_selector`. The promotion logic has been moved inside workers (#38).

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    pitchfork (0.2.0)
+    pitchfork (0.3.0)
       rack (>= 2.0)
       raindrops (~> 0.7)
 

data/docs/CONFIGURATION.md

@@ -238,17 +238,19 @@ application that are used in hooks.
 `pitchfork` also don't attempt to rescue hook errors. Raising from a worker hook will crash the worker,
 and raising from a master hook will bring the whole cluster down.
 
-### `after_promotion`
+### `after_mold_fork`
 
 ```ruby
-after_promotion do |server, mold|
+after_mold_fork do |server, mold|
   Database.disconnect!
+  3.times { GC.start } # promote surviving objects to oldgen
+  GC.compact
 end
 ```
 
-Called in the context of the mold when initially spawned or promotted.
+Called in the context of the mold after it has been spawned.
 
-It's usage is similar to a `before_fork` callback found on other servers
+Its usage is similar to a `before_fork` callback found on other servers
 but it is called once on promotion rather than before forking each worker.
 
 For most protocols connections can be closed after fork, but some
@@ -259,31 +261,18 @@ That is the case for instance of many SQL databases protocols.
 This is also the callback in which memory optimizations, such as
 heap compaction should be done.
 
-### `after_fork`
+### `after_worker_fork`
 
 ```ruby
-after_fork do |server, worker|
-  NetworkClient.disconnect!
+after_worker_fork do |server, worker|
+  NetworkClient.reconnect!
+  BackgroundThread.restart!
 end
 ```
 
 Called in the worker after forking. Generally used to close inherited connections
 or to restart backgrounds threads for libraries that don't do it automatically.
 
-### `after_promotion`
-
-```ruby
-after_promotion do |server, mold|
-  NetworkClient.disconnect!
-  4.times { GC.start } # promote surviving objects to oldgen
-  GC.compact
-end
-```
-
-Called in the worker after it was promoted into a mold. Generally used to shutdown
-open connections and file descriptors, as well as to perform memory optimiations
-such as compacting the heap, trimming memory etc.
-
 ### `after_worker_ready`
 
 Called by a worker process after it has been fully loaded, directly before it

data/docs/DESIGN.md

@@ -28,7 +28,7 @@
 
 * Configuration is purely in Ruby. Ruby is less
   ambiguous than YAML and lets lambdas for
-  before_fork/after_fork hooks be defined inline. An
+  after_worker_fork/after_mold_fork hooks be defined inline. An
   optional, separate config_file may be used to modify supported
   configuration changes.
 

data/docs/FORK_SAFETY.md

@@ -12,16 +12,21 @@ end up shared between the parent and child process. This is never what you
 want, so any code keeping persistent connections should close them either
 before or after the fork happens.
 
-`pitchfork` provide two callbacks in its configuration file to do so:
+Most modern Ruby libraries automatically handle this, it's the case of
+Active Record, redis-rb, dalli, net-http-persistent, etc.
+
+However, for libraries that aren't automatically fork-safe
+`pitchfork` provide two callbacks in its configuration file to manually
+reopen connections and restart threads:
 
 ```ruby
 # pitchfork.conf.rb
 
-before_fork do
+after_mold_fork do
   Sequel::DATABASES.each(&:disconnect)
 end
 
-after_fork do
+after_worker_fork do
   SomeLibary.connection.close
 end
 ```

data/docs/REFORKING.md

@@ -46,50 +46,60 @@ first time, if you take a warmed up worker out of rotation, and use it to fork n
 be shared again, and most of them won't be invalidated anymore.
 
 
-When you start `pitchfork` it forks the desired amount of workers:
+When you start `pitchfork` it forks a `mold` process which loads your application:
 
 ```
 PID   COMMAND
 100   \_ pitchfork master
-101       \_ pitchfork worker[0] (gen:0)
-102       \_ pitchfork worker[1] (gen:0)
-103       \_ pitchfork worker[2] (gen:0)
-104       \_ pitchfork worker[3] (gen:0)
+101       \_ pitchfork (gen:0) mold
 ```
 
-When a reforking is triggered, one of the workers is selected to become a `mold`, and is taken out of rotation.
-When promoted, molds no longer process any incoming HTTP requests, they become inactive:
+Once the `mold` is done loading, the `master` asks it to spawn the desired number of workers:
 
 ```
 PID   COMMAND
 100   \_ pitchfork master
-101       \_ pitchfork mold (gen:1)
-102       \_ pitchfork worker[1] (gen:0)
-103       \_ pitchfork worker[2] (gen:0)
-104       \_ pitchfork worker[3] (gen:0)
+101       \_ pitchfork (gen:0) mold
+102       \_ pitchfork (gen:0) worker[0]
+103       \_ pitchfork (gen:0) worker[1]
+104       \_ pitchfork (gen:0) worker[2]
+105       \_ pitchfork (gen:0) worker[3]
 ```
 
-When a new mold has been promoted, `pitchfork` starts a slow rollout of older workers and replace them with fresh workers
+When a reforking is triggered, one of the workers is selected to fork a new `mold`.
+
+```
+PID   COMMAND
+100   \_ pitchfork master
+101       \_ pitchfork (gen:0) mold
+102       \_ pitchfork (gen:0) worker[0]
+103       \_ pitchfork (gen:0) worker[1]
+104       \_ pitchfork (gen:0) worker[2]
+105       \_ pitchfork (gen:0) worker[3]
+105       \_ pitchfork (gen:1) mold
+```
+
+When that new mold is ready, `pitchfork` terminates the old mold and starts a slow rollout of older workers and replace them with fresh workers
 forked from the mold:
 
 ```
 PID   COMMAND
 100   \_ pitchfork master
-101       \_ pitchfork mold (gen:1)
-105       \_ pitchfork worker[0] (gen:1)
-102       \_ pitchfork worker[1] (gen:0)
-103       \_ pitchfork worker[2] (gen:0)
-104       \_ pitchfork worker[3] (gen:0)
+102       \_ pitchfork (gen:0) worker[0]
+103       \_ pitchfork (gen:0) worker[1]
+104       \_ pitchfork (gen:0) worker[2]
+105       \_ pitchfork (gen:0) worker[3]
+105       \_ pitchfork (gen:1) mold
 ```
 
 ```
 PID   COMMAND
 100   \_ pitchfork master
-101       \_ pitchfork mold (gen:1)
-105       \_ pitchfork worker[0] (gen:1)
-106       \_ pitchfork worker[1] (gen:1)
-103       \_ pitchfork worker[2] (gen:0)
-104       \_ pitchfork worker[3] (gen:0)
+103       \_ pitchfork (gen:0) worker[1]
+104       \_ pitchfork (gen:0) worker[2]
+105       \_ pitchfork (gen:0) worker[3]
+105       \_ pitchfork (gen:1) mold
+106       \_ pitchfork (gen:1) worker[0]
 ```
 
 etc.
@@ -103,7 +113,7 @@ a process tree such as:
 PID   COMMAND
 100   \_ pitchfork master
 101       \_ pitchfork mold (gen:1)
-105          \_ pitchfork worker[0] (gen:1)
+105          \_ pitchfork (gen:1) worker[0]
 ```
 
 However the `pitchfork` master process registers itself as a "child subreaper" via [`PR_SET_CHILD_SUBREAPER`](https://man7.org/linux/man-pages/man2/prctl.2.html).

data/examples/echo.ru

@@ -17,7 +17,6 @@ class EchoBody < Struct.new(:input)
 
 end
 
-use Rack::Chunked
 run lambda { |env|
   /\A100-continue\z/i =~ env['HTTP_EXPECT'] and return [100, {}, []]
   [ 200, { 'content-type' => 'application/octet-stream' },

data/examples/pitchfork.conf.rb

@@ -23,12 +23,7 @@ check_client_connection false
 # local variable to guard against running a hook multiple times
 run_once = true
 
-after_promotion do |server, worker|
-  # the following is highly recommended for Rails
-  # as there's no need for the mold process to hold a connection
-  defined?(ActiveRecord::Base) and
-    ActiveRecord::Base.connection.disconnect!
-
+after_mold_fork do |server, mold|
   # Occasionally, it may be necessary to run non-idempotent code in the
   # master before forking.  Keep in mind the above disconnect! example
   # is idempotent and does not need a guard.
@@ -37,41 +32,8 @@ after_promotion do |server, worker|
     run_once = false # prevent from firing again
   end
 
-  # The following is only recommended for memory/DB-constrained
-  # installations.  It is not needed if your system can house
-  # twice as many worker_processes as you have configured.
-  #
-  # # This allows a new master process to incrementally
-  # # phase out the old master process with SIGTTOU to avoid a
-  # # thundering herd
-  # # when doing a transparent upgrade.  The last worker spawned
-  # # will then kill off the old master process with a SIGQUIT.
-  # old_pid = "#{server.config[:pid]}.oldbin"
-  # if old_pid != server.pid
-  #   begin
-  #     sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
-  #     Process.kill(sig, File.read(old_pid).to_i)
-  #   rescue Errno::ENOENT, Errno::ESRCH
-  #   end
-  # end
-  #
-  # Throttle the master from forking too quickly by sleeping.  Due
-  # to the implementation of standard Unix signal handlers, this
-  # helps (but does not completely) prevent identical, repeated signals
-  # from being lost when the receiving process is busy.
-  # sleep 1
 end
 
-after_fork do |server, worker|
-  # per-process listener ports for debugging/admin/migrations
-  # addr = "127.0.0.1:#{9293 + worker.nr}"
-  # server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)
-
-  # the following is *required* for Rails
-  defined?(ActiveRecord::Base) and
-    ActiveRecord::Base.establish_connection
-
-  # You may also want to check and
-  # restart any other shared sockets/descriptors such as Memcached,
-  # and Redis.
+after_worker_fork do |server, worker|
+  # You may want to check and restart any shared sockets/descriptors
 end