pitchfork 0.4.1 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c0b28e12be38614fe1056d54a89d58882d59100d2982fac86d844574c0c3c196
-  data.tar.gz: 64bf5dfc0d7fc843add452d7737137c96d466f550a95cfd50abe2d0e12b0caad
+  metadata.gz: b3a7c87f91b13b474f5c204a251c380cd091190f3ecfca8ac3107d697e883f12
+  data.tar.gz: '00116558f20d20e4aad0cda6a1866fc7104e3712df27b1668470c52346812258'
 SHA512:
-  metadata.gz: 6bf6a9ffeed660e5cbacf4a3d86c536f28472f9842cf79ad15cd1a1ac2fc7bab351c19040c775e080a2ab4fd0c83607c455b691701d30c308ac9050c968b9ebf
-  data.tar.gz: 789e03e4b35df56c3eb91eb7836aa865f6948e12e959ea196c0fbd757dc6b69be802f005547a6e52ae6ca1932f8a4b0c9cba74a495297d9e138e915ebdc3937f
+  metadata.gz: 2b51ac98fdc2fc2d83a72c6a55c5e26e42996b43bb6e33815e8958c6f5c729f11b17676b9b8bdb098d75b53ebac5bc98114f2bfa864550b3111162824ae6597f
+  data.tar.gz: 5bbf64cf105b20930fd6627001abfb610c9f87ad8510b35a7ab1fcb5c5ccd5437eec011eb6253aefdf87f7d7bc353ac2abbaeb30ac6afad91ddd48fbe81f475c

data/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Unreleased
 
+# 0.6.0
+
+- Expose `Pitchfork::Info.workers_count` and `.live_workers_count` to be consumed by application health checks.
+- Implement `before_worker_exit` callback.
+- Make each mold and worker a process group leader.
+- Get rid of `Pitchfork::PrereadInput`.
+- Add `Pitchfork.shutting_down?` to allow health check endpoints to fail sooner on graceful shutdowns.
+- Treat `TERM` as graceful shutdown rather than quick shutdown.
+- Implement `after_worker_hard_timeout` callback.
+
+# 0.5.0
+
+- Added a soft timeout in addition to the historical Unicorn hard timeout.
+  On soft timeout, the `after_worker_timeout` callback is invoked.
+- Implement `after_request_complete` callback.
+
 # 0.4.1
 
 - Avoid a Rack 3 deprecation warning.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    pitchfork (0.4.1)
+    pitchfork (0.6.0)
       rack (>= 2.0)
       raindrops (~> 0.7)
 

data/benchmark/cow_benchmark.rb

@@ -13,5 +13,5 @@ sleep 3
 puts "Memory Usage:"
 puts Net::HTTP.get(URI(app_url))
 
-Process.kill("TERM", pid)
+Process.kill("INT", pid)
 Process.wait

data/docs/Application_Timeouts.md

@@ -70,5 +70,5 @@ handle network/server failures.
 The `timeout` mechanism in pitchfork is an extreme solution that should
 be avoided whenever possible.
 It will help preserve the platform if your application or a dependency
-has a bug that cause it to either get stuck or two slow, but it is not a
+has a bug that cause it to either get stuck or be too slow, but it is not a
 solution to such bugs, merely a mitigation.

data/docs/CONFIGURATION.md

@@ -173,33 +173,55 @@ The following options may be specified (but are generally not needed):
 ### `timeout`
 
 ```ruby
-timeout 10
+timeout 10, cleanup: 3
 ```
 
-Sets the timeout of worker processes to a number of seconds.
-Workers handling the request/app.call/response cycle taking longer than
-this time period will be forcibly killed (via `SIGKILL`).
+Sets the timeout for worker processes to a number of seconds.
 
-This timeout mecanism shouldn't be routinely relying on, and should
+Note that Pitchfork has two layers of timeout.
+
+A first "soft" timeout will invoke the `after_worker_timeout` from
+within the worker (but from a background thread) and then call `exit`
+to terminate the worker cleanly.
+
+The second "hard" timeout, is the sum of `timeout` and `cleanup`.
+Workers taking longer than this time period to be ready to handle a new
+request will be forcibly killed (via `SIGKILL`).
+
+Neither of these timeout mecanisms should be routinely relied on, and should
 instead be considered as a last line of defense in case you application
 is impacted by bugs causing unexpectedly slow response time, or fully stuck
 processes.
 
+If some of the application endpoints require an unreasonably large timeout,
+rather than to increase the global application timeout, it is possible to
+adjust it on a per request basis via the rack request environment:
+
+```ruby
+class MyMiddleware
+  def call(env)
+    if slow_endpoint?(env)
+      # Give 10 more seconds
+      env["pitchfork.timeout"]&.extend_deadline(10)
+    end
+    @app.call(env)
+  end
+end
+```
+
 Make sure to read the guide on [application timeouts](Application_Timeouts.md).
 
-This configuration defaults to a (too) generous 20 seconds, it is
-highly recommended to set a stricter one based on your application
-profile.
+This configuration defaults to a (too) generous 20 seconds for the soft timeout
+and an extra 2 seconds for the hard timeout. It is highly recommended to set a
+stricter one based on your application profile.
 
-This timeout is enforced by the master process itself and not subject
-to the scheduling limitations by the worker process.
 Due the low-complexity, low-overhead implementation, timeouts of less
 than 3.0 seconds can be considered inaccurate and unsafe.
 
 For running Pitchfork behind nginx, it is recommended to set
 "fail_timeout=0" for in your nginx configuration like this
 to have nginx always retry backends that may have had workers
-SIGKILL-ed due to timeouts.
+exit or be SIGKILL-ed due to timeouts.
 
 ```
    upstream pitchfork_backend {
@@ -284,6 +306,51 @@ after_worker_ready do |server, worker|
 end
 ```
 
+### `after_worker_timeout`
+
+Called by the worker process when the request timeout is elapsed:
+
+```ruby
+after_worker_timeout do |server, worker, timeout_info|
+  timeout_info.copy_thread_variables!
+  timeout_info.thread.kill
+  server.logger.error("Request timed out: #{timeout_info.rack_env.inspect}")
+  $stderr.puts timeout_info.thread.backtrace
+end
+```
+
+Note that this callback is invoked from a different thread. You can access the
+main thread via `timeout_info.thread`, as well as the rack environment via `timeout_info.rack_env`.
+
+If you need to invoke cleanup code that rely on thread local state, you can copy
+that state with `timeout_info.copy_thread_variables!`, but it's best avoided as the
+thread local state could contain thread unsafe objects.
+
+Also note that at this stage, the thread is still alive, if your callback does
+substantial work, you may want to kill the thread.
+
+After the callback is executed the worker will exit with status `0`.
+
+It is recommended not to do slow operations in this callback, but if you
+really have to, make sure to configure the `cleanup` timeout so that the
+callback has time to complete before the "hard" timeout triggers.
+By default the cleanup timeout is 2 seconds.
+
+### `after_worker_hard_timeout`
+
+Called in the master process when a worker hard timeout is elapsed:
+
+```ruby
+after_worker_timeout do |server, worker|
+  $stderr.puts "Worker hard timeout, pid=#{worker.pid}"
+end
+```
+
+Once the callback complete, the worker will be signaled with `SIGKILL`.
+
+This callback being called in an indication that something is preventing the
+soft timeout from working.
+
 ### `after_worker_exit`
 
 Called in the master process after a worker exits.
@@ -297,6 +364,20 @@ after_worker_exit do |server, worker, status|
 end
 ```
 
+### `after_request_complete`
+
+Called in the worker processes after a request has completed.
+
+Can be used for out of band work, or to exit unhealthy workers.
+
+```ruby
+after_request_complete do |server, worker|
+  if something_wrong?
+    exit
+  end
+end
+```
+
 ## Reforking
 
 ### `refork_after`

data/docs/SIGNALS.md

@@ -6,9 +6,9 @@ processes are documented here as well.
 
 ### Master Process
 
-* `INT/TERM` - quick shutdown, kills all workers immediately
+* `INT` - quick shutdown, kills all workers immediately
 
-* `QUIT` - graceful shutdown, waits for workers to finish their
+* `QUIT/TERM` - graceful shutdown, waits for workers to finish their
   current request before finishing.
 
 * `USR2` - trigger a manual refork. A worker is promoted as
@@ -29,10 +29,10 @@ Sending signals directly to the worker processes should not normally be
 needed.  If the master process is running, any exited worker will be
 automatically respawned.
 
-* `INT/TERM` - Quick shutdown, immediately exit.
+* `INT` - Quick shutdown, immediately exit.
   The master process will respawn a worker to replace this one.
   Immediate shutdown is still triggered using kill(2) and not the
   internal pipe as of unicorn 4.8
 
-* `QUIT` - Gracefully exit after finishing the current request.
+* `QUIT/TERM` - Gracefully exit after finishing the current request.
   The master process will respawn a worker to replace this one.

data/lib/pitchfork/configurator.rb

@@ -26,9 +26,15 @@ module Pitchfork
     }
 
     # Default settings for Pitchfork
+    default_logger = Logger.new($stderr)
+    default_logger.formatter = Logger::Formatter.new
+    default_logger.progname = "[Pitchfork]"
+
     DEFAULTS = {
-      :timeout => 20,
-      :logger => Logger.new($stderr),
+      :soft_timeout => 20,
+      :cleanup_timeout => 2,
+      :timeout => 22,
+      :logger => default_logger,
       :worker_processes => 1,
       :after_worker_fork => lambda { |server, worker|
         server.logger.info("worker=#{worker.nr} gen=#{worker.generation} pid=#{$$} spawned")
@@ -36,6 +42,7 @@ module Pitchfork
       :after_mold_fork => lambda { |server, worker|
         server.logger.info("mold gen=#{worker.generation} pid=#{$$} spawned")
       },
+      :before_worker_exit => nil,
       :after_worker_exit => lambda { |server, worker, status|
         m = if worker.nil?
           "repead unknown process (#{status.inspect})"
@@ -53,6 +60,9 @@ module Pitchfork
       :after_worker_ready => lambda { |server, worker|
         server.logger.info("worker=#{worker.nr} gen=#{worker.generation} ready")
       },
+      :after_worker_timeout => nil,
+      :after_worker_hard_timeout => nil,
+      :after_request_complete => nil,
       :early_hints => false,
       :refork_condition => nil,
       :check_client_connection => false,
@@ -131,15 +141,30 @@ module Pitchfork
       set_hook(:after_worker_ready, block_given? ? block : args[0])
     end
 
+    def after_worker_timeout(*args, &block)
+      set_hook(:after_worker_timeout, block_given? ? block : args[0], 3)
+    end
+
+    def after_worker_hard_timeout(*args, &block)
+      set_hook(:after_worker_hard_timeout, block_given? ? block : args[0], 2)
+    end
+
+    def before_worker_exit(*args, &block)
+      set_hook(:before_worker_exit, block_given? ? block : args[0], 2)
+    end
+
     def after_worker_exit(*args, &block)
       set_hook(:after_worker_exit, block_given? ? block : args[0], 3)
     end
 
-    def timeout(seconds)
-      set_int(:timeout, seconds, 3)
-      # POSIX says 31 days is the smallest allowed maximum timeout for select()
-      max = 30 * 60 * 60 * 24
-      set[:timeout] = seconds > max ? max : seconds
+    def after_request_complete(*args, &block)
+      set_hook(:after_request_complete, block_given? ? block : args[0])
+    end
+
+    def timeout(seconds, cleanup: 2)
+      soft_timeout = set_int(:soft_timeout, seconds, 3)
+      cleanup_timeout = set_int(:cleanup_timeout, cleanup, 2)
+      set_int(:timeout, soft_timeout + cleanup_timeout, 5)
     end
 
     def worker_processes(nr)

data/lib/pitchfork/http_server.rb

@@ -1,6 +1,9 @@
 # -*- encoding: binary -*-
 require 'pitchfork/pitchfork_http'
 require 'pitchfork/flock'
+require 'pitchfork/soft_timeout'
+require 'pitchfork/shared_memory'
+require 'pitchfork/info'
 
 module Pitchfork
   # This is the process manager of Pitchfork. This manages worker
@@ -8,13 +11,76 @@ module Pitchfork
   # Listener sockets are started in the master process and shared with
   # forked worker children.
   class HttpServer
+    class TimeoutHandler
+      class Info
+        attr_reader :thread, :rack_env
+
+        def initialize(thread, rack_env)
+          @thread = thread
+          @rack_env = rack_env
+        end
+
+        def copy_thread_variables!
+          current_thread = Thread.current
+          @thread.keys.each do |key|
+            current_thread[key] = @thread[key]
+          end
+          @thread.thread_variables.each do |variable|
+            current_thread.thread_variable_set(variable, @thread.thread_variable_get(variable))
+          end
+        end
+      end
+
+      attr_writer :rack_env, :timeout_request # :nodoc:
+
+      def initialize(server, worker, callback) # :nodoc:
+        @server = server
+        @worker = worker
+        @callback = callback
+        @rack_env = nil
+        @timeout_request = nil
+      end
+
+      def inspect
+        "#<Pitchfork::HttpServer::TimeoutHandler##{object_id}>"
+      end
+
+      def call(original_thread) # :nodoc:
+        begin
+          @server.logger.error("worker=#{@worker.nr} pid=#{@worker.pid} timed out, exiting")
+          if @callback
+            @callback.call(@server, @worker, Info.new(original_thread, @rack_env))
+          end
+        rescue => error
+          Pitchfork.log_error(@server.logger, "after_worker_timeout error", error)
+        end
+        @server.worker_exit(@worker)
+      end
+
+      def finished # :nodoc:
+        @timeout_request.finished
+      end
+
+      def deadline
+        @timeout_request.deadline
+      end
+
+      def extend_deadline(extra_time)
+        extra_time = Integer(extra_time)
+        @worker.deadline += extra_time
+        @timeout_request.extend_deadline(extra_time)
+        self
+      end
+    end
+
     # :stopdoc:
-    attr_accessor :app, :timeout, :worker_processes,
+    attr_accessor :app, :timeout, :soft_timeout, :cleanup_timeout, :worker_processes,
                   :after_worker_fork, :after_mold_fork,
                   :listener_opts, :children,
                   :orig_app, :config, :ready_pipe,
                   :default_middleware, :early_hints
-    attr_writer   :after_worker_exit, :after_worker_ready, :refork_condition
+    attr_writer   :after_worker_exit, :before_worker_exit, :after_worker_ready, :after_request_complete,
+                  :refork_condition, :after_worker_timeout, :after_worker_hard_timeout
 
     attr_reader :logger
     include Pitchfork::SocketHelper
@@ -100,7 +166,9 @@ module Pitchfork
       # list of signals we care about and trap in master.
       @queue_sigs = [
         :QUIT, :INT, :TERM, :USR2, :TTIN, :TTOU ]
-      Worker.preallocate_drops(worker_processes)
+
+      Info.workers_count = worker_processes
+      SharedMemory.preallocate_drops(worker_processes)
     end
 
     # Runs the thing.  Returns self so you can run join on it
@@ -249,7 +317,7 @@ module Pitchfork
       if REFORKING_AVAILABLE && @respawn && @children.molds.empty?
         logger.info("No mold alive, shutting down")
         @exit_status = 1
-        @sig_queue << :QUIT
+        @sig_queue << :TERM
         @respawn = false
       end
 
@@ -269,10 +337,12 @@ module Pitchfork
         end
 
         master_sleep(sleep_time) if sleep
-      when :QUIT # graceful shutdown
-        logger.info "QUIT received, starting graceful shutdown"
+      when :QUIT, :TERM # graceful shutdown
+        SharedMemory.shutting_down!
+        logger.info "#{message} received, starting graceful shutdown"
         return StopIteration
-      when :TERM, :INT # immediate shutdown
+      when :INT # immediate shutdown
+        SharedMemory.shutting_down!
         logger.info "#{message} received, starting immediate shutdown"
         stop(false)
         return StopIteration
@@ -296,7 +366,7 @@ module Pitchfork
         logger.info("mold pid=#{new_mold.pid} gen=#{new_mold.generation} ready")
         old_molds.each do |old_mold|
           logger.info("Terminating old mold pid=#{old_mold.pid} gen=#{old_mold.generation}")
-          old_mold.soft_kill(:QUIT)
+          old_mold.soft_kill(:TERM)
         end
       else
         logger.error("Unexpected message in sig_queue #{message.inspect}")
@@ -307,14 +377,15 @@ module Pitchfork
     # Terminates all workers, but does not exit master process
     def stop(graceful = true)
       @respawn = false
+      SharedMemory.shutting_down!
       wait_for_pending_workers
       self.listeners = []
       limit = Pitchfork.time_now + timeout
       until @children.workers.empty? || Pitchfork.time_now > limit
         if graceful
-          soft_kill_each_child(:QUIT)
+          soft_kill_each_child(:TERM)
         else
-          kill_each_child(:TERM)
+          kill_each_child(:INT)
         end
         if monitor_loop(false) == StopIteration
           return StopIteration
@@ -324,6 +395,17 @@ module Pitchfork
       @promotion_lock.unlink
     end
 
+    def worker_exit(worker)
+      if @before_worker_exit
+        begin
+          @before_worker_exit.call(self, worker)
+        rescue => error
+          Pitchfork.log_error(logger, "before_worker_exit error", error)
+        end
+      end
+      Process.exit
+    end
+
     def rewindable_input
       Pitchfork::HttpParser.input_class.method_defined?(:rewind)
     end
@@ -394,25 +476,39 @@ module Pitchfork
 
     # forcibly terminate all workers that haven't checked in in timeout seconds.  The timeout is implemented using an unlinked File
     def murder_lazy_workers
-      next_sleep = @timeout - 1
       now = Pitchfork.time_now(true)
+      next_sleep = @timeout - 1
+
       @children.workers.each do |worker|
-        tick = worker.tick
-        0 == tick and next # skip workers that haven't processed any clients
-        diff = now - tick
-        tmp = @timeout - diff
-        if tmp >= 0
-          next_sleep > tmp and next_sleep = tmp
+        deadline = worker.deadline
+        if 0 == deadline # worker is idle
           next
+        elsif deadline > now # worker still has time
+          time_left = deadline - now
+          if time_left < next_sleep
+            next_sleep = time_left
+          end
+          next
+        else # worker is out of time
+          next_sleep = 0
+          if worker.mold?
+            logger.error "mold pid=#{worker.pid} timed out, killing"
+          else
+            logger.error "worker=#{worker.nr} pid=#{worker.pid} timed out, killing"
+          end
+
+          if @after_worker_hard_timeout
+            begin
+              @after_worker_hard_timeout.call(self, worker)
+            rescue => error
+              Pitchfork.log_error(@logger, "after_worker_hard_timeout callback", error)
+            end
+          end
+
+          kill_worker(:KILL, worker.pid) # take no prisoners for hard timeout violations
         end
-        next_sleep = 0
-        if worker.mold?
-          logger.error "mold pid=#{worker.pid} timeout (#{diff}s > #{@timeout}s), killing"
-        else
-          logger.error "worker=#{worker.nr} pid=#{worker.pid} timeout (#{diff}s > #{@timeout}s), killing"
-        end
-        kill_worker(:KILL, worker.pid) # take no prisoners for timeout violations
       end
+
       next_sleep <= 0 ? 1 : next_sleep
     end
 
@@ -451,6 +547,7 @@ module Pitchfork
 
         after_fork_internal
         worker_loop(worker)
+        worker_exit(worker)
       end
 
       worker
@@ -509,7 +606,7 @@ module Pitchfork
     def maintain_worker_count
       (off = @children.workers_count - worker_processes) == 0 and return
       off < 0 and return spawn_missing_workers
-      @children.each_worker { |w| w.nr >= worker_processes and w.soft_kill(:QUIT) }
+      @children.each_worker { |w| w.nr >= worker_processes and w.soft_kill(:TERM) }
     end
 
     def restart_outdated_workers
@@ -526,7 +623,7 @@ module Pitchfork
         outdated_workers = @children.workers.select { |w| !w.exiting? && w.generation < @children.mold.generation }
         outdated_workers.each do |worker|
           logger.info("worker=#{worker.nr} pid=#{worker.pid} gen=#{worker.generation} restarting")
-          worker.soft_kill(:QUIT)
+          worker.soft_kill(:TERM)
         end
       end
     end
@@ -577,9 +674,12 @@ module Pitchfork
 
     # once a client is accepted, it is processed in its entirety here
     # in 3 easy steps: read request, call app, write app response
-    def process_client(client)
+    def process_client(client, timeout_handler)
+      env = nil
       @request = Pitchfork::HttpParser.new
       env = @request.read(client)
+      timeout_handler.rack_env = env
+      env["pitchfork.timeout"] = timeout_handler
 
       if early_hints
         env["rack.early_hints"] = lambda do |headers|
@@ -616,6 +716,7 @@ module Pitchfork
       handle_error(client, e)
     ensure
       env["rack.after_reply"].each(&:call) if env
+      timeout_handler.finished
     end
 
     def nuke_listeners!(readers)
@@ -632,7 +733,7 @@ module Pitchfork
     def init_worker_process(worker)
       worker.reset
       worker.register_to_master(@control_socket[1])
-      # we'll re-trap :QUIT later for graceful shutdown iff we accept clients
+      # we'll re-trap :QUIT and :TERM later for graceful shutdown iff we accept clients
       exit_sigs = [ :QUIT, :TERM, :INT ]
       exit_sigs.each { |sig| trap(sig) { exit!(0) } }
       exit!(0) if (@sig_queue & exit_sigs)[0]
@@ -650,6 +751,7 @@ module Pitchfork
       readers = LISTENERS.dup
       readers << worker
       trap(:QUIT) { nuke_listeners!(readers) }
+      trap(:TERM) { nuke_listeners!(readers) }
       readers
     end
 
@@ -658,6 +760,7 @@ module Pitchfork
       after_mold_fork.call(self, mold)
       readers = [mold]
       trap(:QUIT) { nuke_listeners!(readers) }
+      trap(:TERM) { nuke_listeners!(readers) }
       readers
     end
 
@@ -684,7 +787,7 @@ module Pitchfork
 
       while readers[0]
         begin
-          worker.tick = Pitchfork.time_now(true)
+          worker.update_deadline(@timeout)
           while sock = ready.shift
             # Pitchfork::Worker#accept_nonblock is not like accept(2) at all,
             # but that will return false
@@ -697,15 +800,16 @@ module Pitchfork
               when Message
                 worker.update(client)
               else
-                process_client(client)
+                process_client(client, prepare_timeout(worker))
+                @after_request_complete&.call(self, worker)
                 worker.increment_requests_count
               end
-              worker.tick = Pitchfork.time_now(true)
+              worker.update_deadline(@timeout)
             end
           end
 
-          # timeout so we can .tick and keep parent from SIGKILL-ing us
-          worker.tick = Pitchfork.time_now(true)
+          # timeout so we can update .deadline and keep parent from SIGKILL-ing us
+          worker.update_deadline(@timeout)
 
           if @refork_condition && !worker.outdated?
             if @refork_condition.met?(worker, logger)
@@ -754,7 +858,7 @@ module Pitchfork
 
       while readers[0]
         begin
-          mold.tick = Pitchfork.time_now(true)
+          mold.update_deadline(@timeout)
           while sock = ready.shift
             # Pitchfork::Worker#accept_nonblock is not like accept(2) at all,
             # but that will return false
@@ -774,7 +878,7 @@ module Pitchfork
           end
 
           # timeout so we can .tick and keep parent from SIGKILL-ing us
-          mold.tick = Pitchfork.time_now(true)
+          mold.update_deadline(@timeout)
           waiter.get_readers(ready, readers, @timeout * 500) # to milliseconds, but halved
         rescue => e
           Pitchfork.log_error(@logger, "mold loop error", e) if readers[0]
@@ -832,5 +936,11 @@ module Pitchfork
       listeners.each { |addr| listen(addr) }
       raise ArgumentError, "no listeners" if LISTENERS.empty?
     end
+
+    def prepare_timeout(worker)
+      handler = TimeoutHandler.new(self, worker, @after_worker_timeout)
+      handler.timeout_request = SoftTimeout.request(@soft_timeout, handler)
+      handler
+    end
   end
 end

data/lib/pitchfork/info.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require 'pitchfork/shared_memory'
+
+module Pitchfork
+  module Info
+    @workers_count = 0
+
+    class << self
+      attr_accessor :workers_count
+
+      def live_workers_count
+        now = Pitchfork.time_now(true)
+        (0...workers_count).count do |nr|
+          SharedMemory.worker_deadline(nr).value > now
+        end
+      end
+
+      # Returns true if the server is shutting down.
+      # This can be useful to implement health check endpoints, so they
+      # can fail immediately after TERM/QUIT/INT was received by the master
+      # process.
+      # Otherwise they may succeed while Pitchfork is draining requests causing
+      # more requests to be sent.
+      def shutting_down?
+        SharedMemory.shutting_down?
+      end
+    end
+  end
+end

data/lib/pitchfork/shared_memory.rb

@@ -0,0 +1,71 @@
+# frozen_string_literal: true
+
+require 'raindrops'
+
+module Pitchfork
+  module SharedMemory
+    extend self
+
+    PER_DROP = Raindrops::PAGE_SIZE / Raindrops::SIZE
+    CURRENT_GENERATION_OFFSET = 0
+    SHUTDOWN_OFFSET = 1
+    MOLD_TICK_OFFSET = 2
+    WORKER_TICK_OFFSET = 3
+
+    DROPS = [Raindrops.new(PER_DROP)]
+
+    def current_generation
+      DROPS[0][CURRENT_GENERATION_OFFSET]
+    end
+
+    def current_generation=(value)
+      DROPS[0][CURRENT_GENERATION_OFFSET] = value
+    end
+
+    def shutting_down!
+      DROPS[0][SHUTDOWN_OFFSET] = 1
+    end
+
+    def shutting_down?
+      DROPS[0][SHUTDOWN_OFFSET] > 0
+    end
+
+    class Field
+      def initialize(offset)
+        @drop = DROPS.fetch(offset / PER_DROP)
+        @offset = offset % PER_DROP
+      end
+
+      def value
+        @drop[@offset]
+      end
+
+      def value=(value)
+        @drop[@offset] = value
+      end
+    end
+
+    def mold_deadline
+      self[MOLD_TICK_OFFSET]
+    end
+
+    def worker_deadline(worker_nr)
+      self[WORKER_TICK_OFFSET + worker_nr]
+    end
+
+    def [](offset)
+      Field.new(offset)
+    end
+
+    # Since workers are created from another process, we have to
+    # pre-allocate the drops so they are shared between everyone.
+    #
+    # However this doesn't account for TTIN signals that increase the
+    # number of workers, but we should probably remove that feature too.
+    def preallocate_drops(workers_count)
+      0.upto(((WORKER_TICK_OFFSET + workers_count) / PER_DROP.to_f).ceil) do |i|
+        DROPS[i] ||= Raindrops.new(PER_DROP)
+      end
+    end
+  end
+end

data/lib/pitchfork/soft_timeout.rb

@@ -0,0 +1,113 @@
+# frozen_string_literal: true
+
+module Pitchfork
+  # :stopdoc:
+  module SoftTimeout
+    extend self
+
+    CONDVAR = ConditionVariable.new
+    QUEUE = Queue.new
+    QUEUE_MUTEX = Mutex.new
+    TIMEOUT_THREAD_MUTEX = Mutex.new
+    @timeout_thread = nil
+    private_constant :CONDVAR, :QUEUE, :QUEUE_MUTEX, :TIMEOUT_THREAD_MUTEX
+
+    class Request
+      attr_reader :deadline, :thread
+
+      def initialize(thread, timeout, block)
+        @thread = thread
+        @deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
+        @block = block
+
+        @mutex = Mutex.new
+        @done = false # protected by @mutex
+      end
+
+      def extend_deadline(timeout)
+        @deadline += timeout
+        QUEUE_MUTEX.synchronize do
+          CONDVAR.signal
+        end
+        self
+      end
+
+      def done?
+        @mutex.synchronize do
+          @done
+        end
+      end
+
+      def expired?(now)
+        now >= @deadline
+      end
+
+      def interrupt
+        @mutex.synchronize do
+          unless @done
+            begin
+              @block.call(@thread)
+            ensure
+              @done = true
+            end
+          end
+        end
+      end
+
+      def finished
+        @mutex.synchronize do
+          @done = true
+        end
+      end
+    end
+
+    def request(sec, callback)
+      ensure_timeout_thread_created
+      request = Request.new(Thread.current, sec, callback)
+      QUEUE_MUTEX.synchronize do
+        QUEUE << request
+        CONDVAR.signal
+      end
+      request
+    end
+
+    private
+
+    def create_timeout_thread
+      watcher = Thread.new do
+        requests = []
+        while true
+          until QUEUE.empty? and !requests.empty? # wait to have at least one request
+            req = QUEUE.pop
+            requests << req unless req.done?
+          end
+          closest_deadline = requests.min_by(&:deadline).deadline
+
+          now = 0.0
+          QUEUE_MUTEX.synchronize do
+            while (now = Process.clock_gettime(Process::CLOCK_MONOTONIC)) < closest_deadline and QUEUE.empty?
+              CONDVAR.wait(QUEUE_MUTEX, closest_deadline - now)
+            end
+          end
+
+          requests.each do |req|
+            req.interrupt if req.expired?(now)
+          end
+          requests.reject!(&:done?)
+        end
+      end
+      watcher.name = "Pitchfork::Timeout"
+      watcher
+    end
+
+    def ensure_timeout_thread_created
+      unless @timeout_thread and @timeout_thread.alive?
+        TIMEOUT_THREAD_MUTEX.synchronize do
+          unless @timeout_thread and @timeout_thread.alive?
+            @timeout_thread = create_timeout_thread
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pitchfork/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Pitchfork
-  VERSION = "0.4.1"
+  VERSION = "0.6.0"
   module Const
     UNICORN_VERSION = '6.1.0'
   end

data/lib/pitchfork/worker.rb

@@ -1,5 +1,5 @@
 # -*- encoding: binary -*-
-require "raindrops"
+require 'pitchfork/shared_memory'
 
 module Pitchfork
   # This class and its members can be considered a stable interface
@@ -24,7 +24,8 @@ module Pitchfork
       @exiting = false
       @requests_count = 0
       if nr
-        build_raindrops(nr)
+        @deadline_drop = SharedMemory.worker_deadline(nr)
+        self.deadline = 0
       else
         promoted!
       end
@@ -47,7 +48,7 @@ module Pitchfork
     end
 
     def outdated?
-      CURRENT_GENERATION_DROP[0] > @generation
+      SharedMemory.current_generation > @generation
     end
 
     def update(message)
@@ -73,7 +74,7 @@ module Pitchfork
     def finish_promotion(control_socket)
       message = Message::MoldReady.new(@nr, @pid, generation)
       control_socket.sendmsg(message)
-      CURRENT_GENERATION_DROP[0] = @generation
+      SharedMemory.current_generation = @generation
     end
 
     def promote(generation)
@@ -92,8 +93,8 @@ module Pitchfork
     def promoted!
       @mold = true
       @nr = nil
-      @drop_offset = 0
-      @tick_drop = MOLD_DROP
+      @deadline_drop = SharedMemory.mold_deadline
+      self.deadline = 0
       self
     end
 
@@ -167,22 +168,18 @@ module Pitchfork
       super || (!@nr.nil? && @nr == other)
     end
 
+    def update_deadline(timeout)
+      self.deadline = Pitchfork.time_now(true) + timeout
+    end
+
     # called in the worker process
-    def tick=(value) # :nodoc:
-      if mold?
-        MOLD_DROP[0] = value
-      else
-        @tick_drop[@drop_offset] = value
-      end
+    def deadline=(value) # :nodoc:
+      @deadline_drop.value = value
     end
 
     # called in the master process
-    def tick # :nodoc:
-      if mold?
-        MOLD_DROP[0]
-      else
-        @tick_drop[@drop_offset]
-      end
+    def deadline # :nodoc:
+      @deadline_drop.value
     end
 
     def reset
@@ -195,6 +192,7 @@ module Pitchfork
 
     # called in both the master (reaping worker) and worker (SIGQUIT handler)
     def close # :nodoc:
+      self.deadline = 0
       @master.close if @master
       @to_io.close if @to_io
     end
@@ -221,35 +219,10 @@ module Pitchfork
         else
           success = true
         end
-      rescue Errno::EPIPE, Errno::ECONNRESET
+      rescue Errno::EPIPE, Errno::ECONNRESET, Errno::ECONNREFUSED, Errno::ENOTCONN
         # worker will be reaped soon
       end
       success
     end
-
-    MOLD_DROP = Raindrops.new(1)
-    CURRENT_GENERATION_DROP = Raindrops.new(1)
-    PER_DROP = Raindrops::PAGE_SIZE / Raindrops::SIZE
-    TICK_DROPS = []
-
-    class << self
-      # Since workers are created from another process, we have to
-      # pre-allocate the drops so they are shared between everyone.
-      #
-      # However this doesn't account for TTIN signals that increase the
-      # number of workers, but we should probably remove that feature too.
-      def preallocate_drops(workers_count)
-        0.upto(workers_count / PER_DROP) do |i|
-          TICK_DROPS[i] = Raindrops.new(PER_DROP)
-        end
-      end
-    end
-
-    def build_raindrops(drop_nr)
-      drop_index = drop_nr / PER_DROP
-      @drop_offset = drop_nr % PER_DROP
-      @tick_drop = TICK_DROPS[drop_index] ||= Raindrops.new(PER_DROP)
-      @tick_drop[@drop_offset] = 0
-    end
   end
 end

data/lib/pitchfork.rb

@@ -26,9 +26,10 @@ module Pitchfork
   # application dispatch.  This is always raised with an empty backtrace
   # since there is nothing in the application stack that is responsible
   # for client shutdowns/disconnects.  This exception is visible to Rack
-  # applications unless PrereadInput middleware is loaded.  This
-  # is a subclass of the standard EOFError class and applications should
-  # not rescue it explicitly, but rescue EOFError instead.
+  # applications.  This is a subclass of the standard EOFError class and
+  # applications should not rescue it explicitly, but rescue EOFError instead.
+  # Such an error is likely an indication that the reverse proxy in front
+  # of Pitchfork isn't properly buffering requests.
   ClientShutdown = Class.new(EOFError)
 
   BootFailure = Class.new(StandardError)
@@ -120,8 +121,11 @@ module Pitchfork
     end
   end
 
-  def self.clean_fork(&block)
+  def self.clean_fork(setpgid: true, &block)
     if pid = Process.fork
+      if setpgid
+        Process.setpgid(pid, pid) # Make into a group leader
+      end
       return pid
     end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pitchfork
 version: !ruby/object:Gem::Version
-  version: 0.4.1
+  version: 0.6.0
 platform: ruby
 authors:
 - Jean Boussier
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-06-19 00:00:00.000000000 Z
+date: 2023-07-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: raindrops
@@ -101,13 +101,15 @@ files:
 - lib/pitchfork/http_parser.rb
 - lib/pitchfork/http_response.rb
 - lib/pitchfork/http_server.rb
+- lib/pitchfork/info.rb
 - lib/pitchfork/launcher.rb
 - lib/pitchfork/mem_info.rb
 - lib/pitchfork/message.rb
-- lib/pitchfork/preread_input.rb
 - lib/pitchfork/refork_condition.rb
 - lib/pitchfork/select_waiter.rb
+- lib/pitchfork/shared_memory.rb
 - lib/pitchfork/socket_helper.rb
+- lib/pitchfork/soft_timeout.rb
 - lib/pitchfork/stream_input.rb
 - lib/pitchfork/tee_input.rb
 - lib/pitchfork/tmpio.rb

data/lib/pitchfork/preread_input.rb

@@ -1,33 +0,0 @@
-# -*- encoding: binary -*-
-
-module Pitchfork
-  # This middleware is used to ensure input is buffered to memory
-  # or disk (depending on size) before the application is dispatched
-  # by entirely consuming it (from TeeInput) beforehand.
-  #
-  # Usage (in config.ru):
-  #
-  #     require 'pitchfork/preread_input'
-  #     if defined?(Pitchfork)
-  #       use Pitchfork::PrereadInput
-  #     end
-  #     run YourApp.new
-  class PrereadInput
-
-    # :stopdoc:
-    def initialize(app)
-      @app = app
-    end
-
-    def call(env)
-      buf = ""
-      input = env["rack.input"]
-      if input.respond_to?(:rewind)
-        true while input.read(16384, buf)
-        input.rewind
-      end
-      @app.call(env)
-    end
-    # :startdoc:
-  end
-end