pitchfork 0.8.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9c9d47a7cd3604f0807a41882322b4461edf2a990439e152dfa2d94bb731eff7
-  data.tar.gz: 1a38267df9cff0493452fa88c65cb4c9a502643aaf05c8e0b60544f93e9f6e4f
+  metadata.gz: 0f23c677bd64eb8ca7a040cb2d41dab4047e05ae849da16af8e537748bfb6dbe
+  data.tar.gz: 890bac5b67c23d6f0aacf0d9130999cf69ac0bc5a2f1ea06bdd142d67c0606ac
 SHA512:
-  metadata.gz: e8318fc2ae118a7e4a89f65e76e0634d306abd1838f0e4957f638bc870ddd202bc72c006fe3b93b6eec52f1765daea43b24b68eb7b8249b5b21b28a8b9abd51b
-  data.tar.gz: 5f23586cf49e29649496e15577ce7344c477b99878bfd6756f685f2a96ad40fc202d4f3cf97dad8183dd50b2489423c1fcfd48e20303d4eafae6b7f1db586e9b
+  metadata.gz: 39c33bbbe865ba22bebc8b6b1e823f71d0ba6b2787f06af26f0ffe3d4251c781850826fcbcdf1dc13b1734919770e35dc57815d2de057f18d97ba354f247a3e7
+  data.tar.gz: 94e5ae39e129af1fd950c9a77e094bac5bda586f7358115fd8cf25bb3c6ee2079bb4654ebd4e6ed5e9ff498b461559da07d7b44acc0147fcac5c9a4695b7c62c

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Unreleased
 
+# 0.10.0
+
+- Include requests count in workers proctitle.
+
+# 0.9.0
+
+- Implement `spawn_timeout` to protect against bugs causing workers to get stuck before they reach ready state.
+
 # 0.8.0
 
 - Add an `after_monitor_ready` callback, called in the monitor process at end of boot.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    pitchfork (0.8.0)
+    pitchfork (0.10.0)
       rack (>= 2.0)
       raindrops (~> 0.7)
 

data/README.md

@@ -9,13 +9,6 @@ advantage of features in Unix/Unix-like kernels. Slow clients should
 only be served by placing a reverse proxy capable of fully buffering
 both the request and response in between `pitchfork` and slow clients.
 
-## Disclaimer
-
-Until this notice is removed from the README, `pitchfork` should be
-considered experimental. As such it is not encouraged to run it in
-production just yet unless you feel capable of debugging yourself
-any issue that may arise.
-
 ## Features
 
 * Designed for Rack, Linux, fast clients, and ease-of-debugging. We
@@ -40,9 +33,23 @@ any issue that may arise.
   or ports yourself. `pitchfork` can spawn and manage any number of
   worker processes you choose to scale to your backend.
 
+* Adaptative timeout: request timeout can be extended dynamically on a
+  per request basis, which allows to keep a strict overall timeout for
+  most endpoints, but allow a few endpoints to take longer.
+
 * Load balancing is done entirely by the operating system kernel.
   Requests never pile up behind a busy worker process.
 
+## When to Use
+
+Pitchfork isn't inherently better than other Ruby application servers, it mostly
+focus on different tradeoffs.
+
+If you are fine with your current server, it's best to stick with it.
+
+If there is a problem you are trying to solve, please read the
+[migration guide](docs/WHY_MIGRATE.md) first.
+
 ## Requirements
 
 Ruby(MRI) Version 2.5 and above.

data/docs/CONFIGURATION.md

@@ -238,6 +238,19 @@ exit or be SIGKILL-ed due to timeouts.
 See https://nginx.org/en/docs/http/ngx_http_upstream_module.html
 for more details on nginx upstream configuration.
 
+### `spawn_timeout`
+
+```ruby
+timeout 5
+```
+
+Sets the timeout for a newly spawned worker to be ready after being spawned.
+
+This timeout is a safeguard against various low-level fork safety bugs that could cause
+a process to dead-lock.
+
+The default of `10` seconds is quite generous and likely doesn't need to be adjusted.
+
 ### `logger`
 
 ```ruby

data/docs/REFORKING.md

@@ -66,7 +66,10 @@ PID   COMMAND
 105       \_ pitchfork (gen:0) worker[3]
 ```
 
-When a reforking is triggered, one of the workers is selected to fork a new `mold`.
+As the diagram shows, while workers are forked from the mold, they become children of the master process.
+We'll see how does that work [later](#forking-sibling-processes).
+
+When a reforking is triggered, one of the workers is selected to fork a new `mold`:
 
 ```
 PID   COMMAND
@@ -79,6 +82,9 @@ PID   COMMAND
 105       \_ pitchfork (gen:1) mold
 ```
 
+Again, while the mold was forked from a worker, it becomes a child of the master process.
+We'll see how does that work [later](#forking-sibling-processes).
+
 When that new mold is ready, `pitchfork` terminates the old mold and starts a slow rollout of older workers and replace them with fresh workers
 forked from the mold:
 
@@ -104,7 +110,7 @@ PID   COMMAND
 
 etc.
 
-### Forking Sibling Processes 
+### Forking Sibling Processes
 
 Normally on unix systems, when calling `fork(2)`, the newly created process is a child of the original one, so forking from the mold should create
 a process tree such as:
@@ -119,5 +125,8 @@ PID   COMMAND
 However the `pitchfork` master process registers itself as a "child subreaper" via [`PR_SET_CHILD_SUBREAPER`](https://man7.org/linux/man-pages/man2/prctl.2.html).
 This means any descendant process that is orphaned will be re-parented as a child of the master rather than a child of the init process (pid 1).
 
-With this in mind, the mold fork twice to create an orphaned process that will get re-attached to the master, effectively forking a sibling rather than a child.
-The need for `PR_SET_CHILD_SUBREAPER` is the main reason why reforking is only available on Linux. 
+With this in mind, the mold forks twice to create an orphaned process that will get re-attached to the master,
+effectively forking a sibling rather than a child. Similarly, workers do the same when forking new molds.
+This technique eases killing previous generations of molds and workers.
+
+The need for `PR_SET_CHILD_SUBREAPER` is the main reason why reforking is only available on Linux.

data/docs/WHY_MIGRATE.md

@@ -0,0 +1,93 @@
+# Why migrate to Pitchfork?
+
+First and foremost, if you don't have any specific problem with your current server, then don't.
+
+Pitchfork isn't a silver bullet, it's a very opinionated software that focus on very specific tradeoffs,
+that are different from other servers.
+
+## Coming from Unicorn
+
+### Why Migrate?
+
+#### Adaptative timeout
+
+Pitchfork allows to extend the request timeout on a per request basis,
+this can be helpful when trying to reduce the global request timeout
+to a saner value. You can enforce a stricter value, and extend it
+in the minority of offending endpoints.
+
+#### Memory Usage - Reforking
+
+If you are unsatisfied with Unicorn memory usage, but threaded Puma isn't an option
+for you, then Pitchfork may be an option if you are able to enable reforking.
+
+However be warned that making an application fork safe can be non-trivial,
+and mistakes can lead to critical bugs.
+
+#### Rack 3
+
+As of Unicorn `6.1.0`, Rack 3 isn't yet supported by Unicorn.
+
+Pitchfork is compatible with Rack 3.
+
+### Why Not Migrate?
+
+#### Reduced Features
+
+While Pitchfork started as a fork of Unicorn, many features such as daemonization,
+pid file management, hot reload have been stripped.
+
+Pitchfork only kept features that makes sense in a containerized world.
+
+## Coming from Puma
+
+Generally speaking, compared to (threaded) Puma, Pitchfork *may* offer better latency and isolation at the expense of throughput.
+
+### Why Migrate?
+
+#### Latency
+
+If you suspect your application is subject to contention on the GVL or some other in-process shared resources,
+then Pitchfork may offer improved latency.
+
+It is however heavily recommended to first confirm this suspicion with profiling
+tools such as [gvltools](https://github.com/Shopify/gvltools).
+
+If you application isn't subject to in-process contention, Pitchfork is unlikely to improve latency.
+
+#### Out of Band Garbage Collection
+
+Another advantage of only processing a single request per process is that
+[it allows to periodically trigger garbage collection when the worker isn't processing any request](https://shopify.engineering/adventures-in-garbage-collection).
+
+This can significantly improve tail latency at the expense of throughput.
+
+#### Resiliency and Isolation
+
+Since Pitchfork workers have their own address space and only process one request at a time
+it makes it much harder for one faulty request to impact another.
+
+Even if a bug causes Ruby to crash, only the request that triggered the bug will be impacted.
+
+If a bug causes Ruby to hang, the monitor process will SIGKILL the worker and the capacity will be
+reclaimed.
+
+This makes Pitchfork more resilient to some classes of bugs.
+
+#### Thread Safety
+
+Pitchfork doesn't require applications to be thread-safe. That is probably the worst reason
+to migrate though.
+
+### Why Not Migrate?
+
+#### Memory Usage
+
+Without reforking enabled Pitchfork will without a doubt use more memory than threaded Puma.
+
+With reforking enabled, results will vary based on the application profile and the number of Puma threads,
+but should be in the same ballpark, sometimes better, but likely worse, this depends on many variables and
+can't really be predicted.
+
+However be warned that [making an application fork safe](FORK_SAFETY.md) can be non-trivial,
+and mistakes can lead to critical bugs.

data/lib/pitchfork/children.rb

@@ -107,6 +107,33 @@ module Pitchfork
       @workers.each_value(&block)
     end
 
+    def soft_kill_all(sig)
+      each do |child|
+        child.soft_kill(sig)
+      end
+    end
+
+    def hard_kill(sig, child)
+      child.hard_kill(sig)
+    rescue Errno::ESRCH
+      reap(child.pid)
+      child.close
+    end
+
+    def hard_kill_all(sig)
+      each do |child|
+        hard_kill(sig, child)
+      end
+    end
+
+    def hard_timeout(child)
+      child.hard_timeout!
+    rescue Errno::ESRCH
+      reap(child.pid)
+      child.close
+      true
+    end
+
     def workers
       @workers.values
     end

data/lib/pitchfork/configurator.rb

@@ -33,6 +33,7 @@ module Pitchfork
     DEFAULTS = {
       :soft_timeout => 20,
       :cleanup_timeout => 2,
+      :spawn_timeout => 10,
       :timeout => 22,
       :logger => default_logger,
       :worker_processes => 1,
@@ -174,6 +175,10 @@ module Pitchfork
       set_int(:timeout, soft_timeout + cleanup_timeout, 5)
     end
 
+    def spawn_timeout(seconds)
+      set_int(:spawn_timeout, seconds, 1)
+    end
+
     def worker_processes(nr)
       set_int(:worker_processes, nr, 1)
     end

data/lib/pitchfork/http_server.rb

@@ -74,7 +74,7 @@ module Pitchfork
     end
 
     # :stopdoc:
-    attr_accessor :app, :timeout, :soft_timeout, :cleanup_timeout, :worker_processes,
+    attr_accessor :app, :timeout, :soft_timeout, :cleanup_timeout, :spawn_timeout, :worker_processes,
                   :after_worker_fork, :after_mold_fork,
                   :listener_opts, :children,
                   :orig_app, :config, :ready_pipe,
@@ -396,15 +396,15 @@ module Pitchfork
       limit = Pitchfork.time_now + timeout
       until @children.workers.empty? || Pitchfork.time_now > limit
         if graceful
-          soft_kill_each_child(:TERM)
+          @children.soft_kill_all(:TERM)
         else
-          kill_each_child(:INT)
+          @children.hard_kill_all(:INT)
         end
         if monitor_loop(false) == StopIteration
           return StopIteration
         end
       end
-      kill_each_child(:KILL)
+      @children.hard_kill_all(:KILL)
       @promotion_lock.unlink
     end
 
@@ -506,25 +506,28 @@ module Pitchfork
           next
         else # worker is out of time
           next_sleep = 0
-          if worker.mold?
-            logger.error "mold pid=#{worker.pid} timed out, killing"
-          else
-            logger.error "worker=#{worker.nr} pid=#{worker.pid} timed out, killing"
-          end
+          hard_timeout(worker)
+        end
+      end
 
-          if @after_worker_hard_timeout
-            begin
-              @after_worker_hard_timeout.call(self, worker)
-            rescue => error
-              Pitchfork.log_error(@logger, "after_worker_hard_timeout callback", error)
-            end
-          end
+      next_sleep <= 0 ? 1 : next_sleep
+    end
 
-          kill_worker(:KILL, worker.pid) # take no prisoners for hard timeout violations
+    def hard_timeout(worker)
+      if @after_worker_hard_timeout
+        begin
+          @after_worker_hard_timeout.call(self, worker)
+        rescue => error
+          Pitchfork.log_error(@logger, "after_worker_hard_timeout callback", error)
         end
       end
 
-      next_sleep <= 0 ? 1 : next_sleep
+      if worker.mold?
+        logger.error "mold pid=#{worker.pid} timed out, killing"
+      else
+        logger.error "worker=#{worker.nr} pid=#{worker.pid} timed out, killing"
+      end
+      @children.hard_timeout(worker) # take no prisoners for hard timeout violations
     end
 
     def trigger_refork
@@ -556,6 +559,10 @@ module Pitchfork
     def spawn_worker(worker, detach:)
       logger.info("worker=#{worker.nr} gen=#{worker.generation} spawning...")
 
+      # We set the deadline before spawning the child so that if for some
+      # reason it gets stuck before reaching the worker loop,
+      # the monitor process will kill it.
+      worker.update_deadline(@spawn_timeout)
       Pitchfork.fork_sibling do
         worker.pid = Process.pid
 
@@ -693,12 +700,12 @@ module Pitchfork
 
     # once a client is accepted, it is processed in its entirety here
     # in 3 easy steps: read request, call app, write app response
-    def process_client(client, timeout_handler)
+    def process_client(client, worker, timeout_handler)
       env = nil
       @request = Pitchfork::HttpParser.new
       env = @request.read(client)
 
-      proc_name status: "processing: #{env["PATH_INFO"]}"
+      proc_name status: "requests: #{worker.requests_count}, processing: #{env["PATH_INFO"]}"
 
       timeout_handler.rack_env = env
       env["pitchfork.timeout"] = timeout_handler
@@ -831,7 +838,7 @@ module Pitchfork
               when Message
                 worker.update(client)
               else
-                request_env = process_client(client, prepare_timeout(worker))
+                request_env = process_client(client, worker, prepare_timeout(worker))
                 @after_request_complete&.call(self, worker, request_env)
                 worker.increment_requests_count
               end
@@ -844,6 +851,7 @@ module Pitchfork
 
           if @refork_condition && Info.fork_safe? && !worker.outdated?
             if @refork_condition.met?(worker, logger)
+              proc_name status: "requests: #{worker.requests_count}, spawning mold"
               if spawn_mold(worker.generation)
                 logger.info("Refork condition met, promoting ourselves")
               end
@@ -851,7 +859,7 @@ module Pitchfork
             end
           end
 
-          proc_name status: "waiting"
+          proc_name status: "requests: #{worker.requests_count}, waiting"
           waiter.get_readers(ready, readers, @timeout * 500) # to milliseconds, but halved
         rescue => e
           Pitchfork.log_error(@logger, "listen loop error", e) if readers[0]
@@ -926,15 +934,6 @@ module Pitchfork
       worker = @children.reap(wpid) and worker.close rescue nil
     end
 
-    # delivers a signal to each worker
-    def kill_each_child(signal)
-      @children.each { |w| kill_worker(signal, w.pid) }
-    end
-
-    def soft_kill_each_child(signal)
-      @children.each { |worker| worker.soft_kill(signal) }
-    end
-
     # returns an array of string names for the given listener array
     def listener_names(listeners = LISTENERS)
       listeners.map { |io| sock_name(io) }

data/lib/pitchfork/info.rb

@@ -6,14 +6,35 @@ module Pitchfork
   module Info
     @workers_count = 0
     @fork_safe = true
-    @kept_ios = ObjectSpace::WeakMap.new
+
+    class WeakSet # :nodoc
+      def initialize
+        @map = ObjectSpace::WeakMap.new
+      end
+
+      if RUBY_VERSION < "2.7"
+        def <<(object)
+          @map[object] = object
+        end
+      else
+        def <<(object)
+          @map[object] = true
+        end
+      end
+
+      def each(&block)
+        @map.each_key(&block)
+      end
+    end
+    
+    @kept_ios = WeakSet.new
 
     class << self
       attr_accessor :workers_count
 
       def keep_io(io)
         raise ArgumentError, "#{io.inspect} doesn't respond to :to_io" unless io.respond_to?(:to_io)
-        @kept_ios[io] = io
+        @kept_ios << io
         io
       end
 
@@ -22,9 +43,9 @@ module Pitchfork
       end
 
       def close_all_ios!
-        ignored_ios = [$stdin, $stdout, $stderr]
+        ignored_ios = [$stdin, $stdout, $stderr, STDIN, STDOUT, STDERR].uniq.compact
 
-        @kept_ios.each_value do |io_like|
+        @kept_ios.each do |io_like|
           ignored_ios << (io_like.is_a?(IO) ? io_like : io_like.to_io)
         end
 

data/lib/pitchfork/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Pitchfork
-  VERSION = "0.8.0"
+  VERSION = "0.10.0"
   module Const
     UNICORN_VERSION = '6.1.0'
   end

data/lib/pitchfork/worker.rb

@@ -139,6 +139,14 @@ module Pitchfork
       success
     end
 
+    def hard_kill(sig)
+      Process.kill(sig, pid)
+    end
+
+    def hard_timeout!
+      hard_kill(:KILL)
+    end
+
     # this only runs when the Rack app.call is not running
     # act like a listener
     def accept_nonblock(exception: nil) # :nodoc:

data/lib/pitchfork.rb

@@ -204,8 +204,13 @@ module Pitchfork
         # or the master to be PID 1.
         if middle_pid = FORK_LOCK.synchronize { Process.fork } # parent
           # We need to wait(2) so that the middle process doesn't end up a zombie.
-          Process.wait(middle_pid)
+          # The process only call fork again an exit so it should be pretty fast.
+          # However it might need to execute some `Process._fork` or `at_exit` callbacks,
+          # so it case it takes more than 5 seconds to exit, we kill it with SIGBUS
+          # to produce a crash report, as this is indicative of a nasty bug.
+          process_wait_with_timeout(middle_pid, 5, :BUS)
         else # first child
+          Process.setproctitle("<pitchfork fork_sibling>")
           clean_fork(&block) # detach into a grand child
           exit
         end
@@ -216,6 +221,18 @@ module Pitchfork
       nil # it's tricky to return the PID
     end
 
+    def process_wait_with_timeout(pid, timeout, timeout_signal = :KILL)
+      (timeout * 200).times do
+        status = Process.wait(pid, Process::WNOHANG)
+        return status if status
+        sleep 0.005 # 200 * 5ms => 1s
+      end
+
+      # The process didn't exit in the allotted time, so we kill it.
+      Process.kill(timeout_signal, pid)
+      Process.wait(pid)
+    end
+
     def time_now(int = false)
       Process.clock_gettime(Process::CLOCK_MONOTONIC, int ? :second : :float_second)
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pitchfork
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 0.10.0
 platform: ruby
 authors:
 - Jean Boussier
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-09-08 00:00:00.000000000 Z
+date: 2023-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: raindrops
@@ -72,6 +72,7 @@ files:
 - docs/REFORKING.md
 - docs/SIGNALS.md
 - docs/TUNING.md
+- docs/WHY_MIGRATE.md
 - examples/constant_caches.ru
 - examples/echo.ru
 - examples/hello.ru