rack-timeout 0.5.2 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: a13210b3a4119f8fb5d6c4bc6f6c48ab24fbddcd
-  data.tar.gz: 37718bb72a7a86a3c5e525e37546bf84891100df
+SHA256:
+  metadata.gz: 45a8b583f5c8ec73b0659348e53083fd449d1ae732c020c45ab3decfd4d7c913
+  data.tar.gz: 832b443cc5678f0c55df7a8c741dc2f5304e024da77f021ecbaa352c03279e51
 SHA512:
-  metadata.gz: 5ed18160165e8511b56d7627dcc82f129162c4dbaeacaaea3b25cef1b66ec72b601a33bf9413b39bd24c434972278f3dd54b508d7452a3fa934607c034c33a1c
-  data.tar.gz: 4c9bf93aefac52b1523a6619b8ff0b9b2e0ee8dd7a7cc818df3ebc6c8ca80354e11e5119cb75ee172922abdef3d7784afe9090ec52521fbf333cd86020479f38
+  metadata.gz: 2279854e2ca96bc0fa0c9e6fe4a67a4d217f79df5840b59cf36b97518ddab7823a134c15fe1e527ce1a24972ee9113825bb58d52520accdcb7dc2a3d8147cb16
+  data.tar.gz: 00651f0c2e2449d490e88db4cf7d899f1287f22bc531fc4c22270024790635b552119572847de65a1933a5b7e4e6514c530a33b0689c02d6aeb34e73de392245

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+## 0.6.0
+
+- Allow sending SIGTERM to workers on timeout (https://github.com/sharpstone/rack-timeout/pull/157)
+
 0.5.2
 =====
 - Rails 6 support (#147)

data/README.md

@@ -47,7 +47,7 @@ stack `Rack::Timeout` gets inserted.
 
 ```ruby
 # Gemfile
-gem "rack-timeout", require:"rack/timeout/base"
+gem "rack-timeout", require: "rack/timeout/base"
 ```
 
 ```ruby
@@ -55,7 +55,7 @@ gem "rack-timeout", require:"rack/timeout/base"
 
 # insert middleware wherever you want in the stack, optionally pass
 # initialization arguments, or use environment variables
-Rails.application.config.middleware.insert_before Rack::Runtime, Rack::Timeout, service_timeout: 5
+Rails.application.config.middleware.insert_before Rack::Runtime, Rack::Timeout, service_timeout: 15
 ```
 
 ### Sinatra and other Rack apps
@@ -67,7 +67,7 @@ require "rack-timeout"
 # Call as early as possible so rack-timeout runs before all other middleware.
 # Setting service_timeout or `RACK_TIMEOUT_SERVICE_TIMEOUT` environment
 # variable is recommended. If omitted, defaults to 15 seconds.
-use Rack::Timeout, service_timeout: 5
+use Rack::Timeout, service_timeout: 15
 ```
 
 Configuring
@@ -81,6 +81,7 @@ service_timeout:   15     # RACK_TIMEOUT_SERVICE_TIMEOUT
 wait_timeout:      30     # RACK_TIMEOUT_WAIT_TIMEOUT
 wait_overtime:     60     # RACK_TIMEOUT_WAIT_OVERTIME
 service_past_wait: false  # RACK_TIMEOUT_SERVICE_PAST_WAIT
+term_on_timeout:   false  # RACK_TIMEOUT_TERM_ON_TIMEOUT
 ```
 
 These settings can be overriden during middleware initialization or
@@ -88,7 +89,7 @@ environment variables `RACK_TIMEOUT_*` mentioned above. Middleware
 parameters take precedence:
 
 ```ruby
-use Rack::Timeout, service_timeout: 5, wait_timeout: false
+use Rack::Timeout, service_timeout: 15, wait_timeout: 30
 ```
 
 For more on these settings, please see [doc/settings](doc/settings.md).

data/doc/risks.md

@@ -26,7 +26,6 @@ That said, it's something to be aware of, and may explain some eerie wonkiness s
 [broken-timeout]: http://headius.blogspot.de/2008/02/rubys-threadraise-threadkill-timeoutrb.html
 [handle-interrupt]: http://www.ruby-doc.org/core-2.1.3/Thread.html#method-c-handle_interrupt
 
-
 ### Time Out Early and Often
 
 Because of the aforementioned issues, it's recommended you set library-specific timeouts and leave Rack::Timeout as a last resort measure. Library timeouts will generally take care of IO issues and abort the operation safely. See [The Ultimate Guide to Ruby Timeouts][ruby-timeouts].

data/doc/settings.md

@@ -47,3 +47,55 @@ This extra time is called *wait overtime* and can be set via `wait_overtime`. It
 Keep in mind that Heroku [recommends][uploads] uploading large files directly to S3, so as to prevent the dyno from being blocked for too long and hence unable to handle further incoming requests.
 
 [uploads]: https://devcenter.heroku.com/articles/s3#file-uploads
+
+### Term on Timeout
+
+If your application timeouts fire frequently then [they can cause your application to enter a corrupt state](https://www.schneems.com/2017/02/21/the-oldest-bug-in-ruby-why-racktimeout-might-hose-your-server/). One option for resetting that bad state is to restart the entire process. If you are running in an environment with multiple processes (such as `puma -w 2`) then when a process is sent a `SIGTERM` it will exit. The webserver then knows how to restart the process. For more information on process restart behavior see:
+
+- [Ruby Application Restart Behavior](https://devcenter.heroku.com/articles/what-happens-to-ruby-apps-when-they-are-restarted)
+- [License to SIGKILL](https://www.sitepoint.com/license-to-sigkill/)
+
+**Puma SIGTERM behavior** When a Puma worker receives a `SIGTERM` it will begin to shut down, but not exit right away. It stops accepting new requests and waits for any existing requests to finish before fully shutting down. This means that only the request that experiences a timeout will be interupted, all other in-flight requests will be allowed to run until they return or also are timed out.
+
+After the worker process exists will Puma's parent process know to boot a replacement worker. While one process is restarting, another can still serve requests (if you have more than 1 worker process per server/dyno). Between when a process exits and when a new process boots, there will be a reduction in throughput. If all processes are restarting, then incoming requests will be blocked while new processes boot.
+
+**How to enable** To enable this behavior you can set `term_on_timeout: 1` to an integer value. If you set it to zero or one, then the first time the process encounters a timeout, it will receive a SIGTERM.
+
+To enable on Heroku run:
+
+```
+$ heroku config:set RACK_TIMEOUT_TERM_ON_TIMEOUT=1
+```
+
+**Caution** If you use this setting inside of a webserver without enabling multi-process mode, then it will exit the entire server when it fires:
+
+- ✅ `puma -w 2 -t 5` This is OKAY
+- ❌ `puma -t 5` This is NOT OKAY
+
+If you're using a `config/puma.rb` file then make sure you are calling `workers` configuration DSL. You should see multiple workers when the server boots:
+
+```
+[3922] Puma starting in cluster mode...
+[3922] * Version 4.3.0 (ruby 2.6.5-p114), codename: Mysterious Traveller
+[3922] * Min threads: 0, max threads: 16
+[3922] * Environment: development
+[3922] * Process workers: 2
+[3922] * Phased restart available
+[3922] * Listening on tcp://0.0.0.0:9292
+[3922] Use Ctrl-C to stop
+[3922] - Worker 0 (pid: 3924) booted, phase: 0
+[3922] - Worker 1 (pid: 3925) booted, phase: 0
+```
+
+> ✅ Notice how it says it is booting in "cluster mode" and how it gives PIDs for two worker processes at the bottom.
+
+**How to decide the term_on_timeout value** If you set to a higher value such as `5` then rack-timeout will wait until the process has experienced five timeouts before restarting the process. Setting this value to a higher number means the application restarts processes less frequently, so throughput will be less impacted. If you set it to too high of a number, then the underlying issue of the application being put into a bad state will not be effectively mitigated.
+
+
+**How do I know when a process is being restarted by rack-timeout?** This exception error should be visible in the logs:
+
+```
+Request ran for longer than 1000ms, sending SIGTERM to process 3925
+```
+
+> Note: Since the worker waits for all in-flight requests to finish (with puma) you may see multiple SIGTERMs to the same PID before it exits, this means that multiple requests timed out.

data/lib/rack/timeout/core.rb

@@ -30,6 +30,7 @@ module Rack
       :service,   # time rack spent processing the request (updated ~ every second)
       :timeout,   # the actual computed timeout to be used for this request
       :state,     # the request's current state, see VALID_STATES below
+      :term,
     ) {
       def ms(k)   # helper method used for formatting values in milliseconds
         "%.fms" % (self[k] * 1000) if self[k]
@@ -52,6 +53,8 @@ module Rack
       when nil   ; read_timeout_property default, default
       when false ; false
       when 0     ; false
+      when String
+        read_timeout_property value.to_i, default
       else
         value.is_a?(Numeric) && value > 0 or raise ArgumentError, "value #{value.inspect} should be false, zero, or a positive number."
         value
@@ -62,13 +65,21 @@ module Rack
       :service_timeout,   # How long the application can take to complete handling the request once it's passed down to it.
       :wait_timeout,      # How long the request is allowed to have waited before reaching rack. If exceeded, the request is 'expired', i.e. dropped entirely without being passed down to the application.
       :wait_overtime,     # Additional time over @wait_timeout for requests with a body, like POST requests. These may take longer to be received by the server before being passed down to the application, but should not be expired.
-      :service_past_wait  # when false, reduces the request's computed timeout from the service_timeout value if the complete request lifetime (wait + service) would have been longer than wait_timeout (+ wait_overtime when applicable). When true, always uses the service_timeout value. we default to false under the assumption that the router would drop a request that's not responded within wait_timeout, thus being there no point in servicing beyond seconds_service_left (see code further down) up until service_timeout.
+      :service_past_wait, # when false, reduces the request's computed timeout from the service_timeout value if the complete request lifetime (wait + service) would have been longer than wait_timeout (+ wait_overtime when applicable). When true, always uses the service_timeout value. we default to false under the assumption that the router would drop a request that's not responded within wait_timeout, thus being there no point in servicing beyond seconds_service_left (see code further down) up until service_timeout.
+      :term_on_timeout
 
-    def initialize(app, service_timeout:nil, wait_timeout:nil, wait_overtime:nil, service_past_wait:"not_specified")
+    def initialize(app, service_timeout:nil, wait_timeout:nil, wait_overtime:nil, service_past_wait:"not_specified", term_on_timeout: nil)
+      @term_on_timeout   = read_timeout_property term_on_timeout, ENV.fetch("RACK_TIMEOUT_TERM_ON_TIMEOUT", false)
       @service_timeout   = read_timeout_property service_timeout, ENV.fetch("RACK_TIMEOUT_SERVICE_TIMEOUT", 15).to_i
       @wait_timeout      = read_timeout_property wait_timeout,    ENV.fetch("RACK_TIMEOUT_WAIT_TIMEOUT", 30).to_i
       @wait_overtime     = read_timeout_property wait_overtime,   ENV.fetch("RACK_TIMEOUT_WAIT_OVERTIME", 60).to_i
       @service_past_wait = service_past_wait == "not_specified" ? ENV.fetch("RACK_TIMEOUT_SERVICE_PAST_WAIT", false).to_s != "false" : service_past_wait
+
+      Thread.main['RACK_TIMEOUT_COUNT'] ||= 0
+      if @term_on_timeout
+        raise "term_on_timeout must be an integer but is #{@term_on_timeout.class}: #{@term_on_timeout}" unless @term_on_timeout.is_a?(Numeric)
+        raise "Current Runtime does not support processes" unless ::Process.respond_to?(:fork)
+      end
       @app = app
     end
 
@@ -90,7 +101,9 @@ module Rack
         seconds_waited          = 0 if seconds_waited < 0                  # make up for potential time drift between the routing server and the application server
         final_wait_timeout      = wait_timeout + effective_overtime        # how long the request will be allowed to have waited
         seconds_service_left    = final_wait_timeout - seconds_waited      # first calculation of service timeout (relevant if request doesn't get expired, may be overriden later)
-        info.wait, info.timeout = seconds_waited, final_wait_timeout       # updating the info properties; info.timeout will be the wait timeout at this point
+        info.wait               = seconds_waited                           # updating the info properties; info.timeout will be the wait timeout at this point
+        info.timeout            = final_wait_timeout
+
         if seconds_service_left <= 0 # expire requests that have waited for too long in the queue (as they are assumed to have been dropped by the web server / routing layer at this point)
           RT._set_state! env, :expired
           raise RequestExpiryError.new(env), "Request older than #{info.ms(:timeout)}."
@@ -103,7 +116,7 @@ module Rack
       # compute actual timeout to be used for this request; if service_past_wait is true, this is just service_timeout. If false (the default), and wait time was determined, we'll use the shortest value between seconds_service_left and service_timeout. See comment above at service_past_wait for justification.
       info.timeout = service_timeout # nice and simple, when service_past_wait is true, not so much otherwise:
       info.timeout = seconds_service_left if !service_past_wait && seconds_service_left && seconds_service_left > 0 && seconds_service_left < service_timeout
-
+      info.term    = term_on_timeout
       RT._set_state! env, :ready                            # we're good to go, but have done nothing yet
 
       heartbeat_event = nil                                 # init var so it's in scope for following proc
@@ -116,7 +129,22 @@ module Rack
 
       timeout = RT::Scheduler::Timeout.new do |app_thread|  # creates a timeout instance responsible for timing out the request. the given block runs if timed out
         register_state_change.call :timed_out
-        app_thread.raise(RequestTimeoutException.new(env), "Request #{"waited #{info.ms(:wait)}, then " if info.wait}ran for longer than #{info.ms(:timeout)}")
+
+        message = "Request "
+        message << "waited #{info.ms(:wait)}, then " if info.wait
+        message << "ran for longer than #{info.ms(:timeout)} "
+        if term_on_timeout
+          Thread.main['RACK_TIMEOUT_COUNT'] += 1
+
+          if Thread.main['RACK_TIMEOUT_COUNT'] >= @term_on_timeout
+            message << ", sending SIGTERM to process #{Process.pid}"
+            Process.kill("SIGTERM", Process.pid)
+          else
+            message << ", #{Thread.main['RACK_TIMEOUT_COUNT']}/#{term_on_timeout} timeouts allowed before SIGTERM for process #{Process.pid}"
+          end
+        end
+
+        app_thread.raise(RequestTimeoutException.new(env), message)
       end
 
       response = timeout.timeout(info.timeout) do           # perform request with timeout
@@ -191,6 +219,5 @@ module Rack
     def self.notify_state_change_observers(env)
       @state_change_observers.values.each { |observer| observer.call(env) }
     end
-
   end
 end

data/lib/rack/timeout/logger.rb

@@ -35,5 +35,4 @@ module Rack::Timeout::Logger
     @level      = new_level  || ::Logger::INFO
     self.logger = ::Rack::Timeout::StateChangeLoggingObserver.mk_logger(device, level)
   end
-
 end

data/lib/rack/timeout/logging-observer.rb

@@ -48,9 +48,9 @@ class Rack::Timeout::StateChangeLoggingObserver
       s << " wait="    << info.ms(:wait)    if info.wait
       s << " timeout=" << info.ms(:timeout) if info.timeout
       s << " service=" << info.ms(:service) if info.service
+      s << " term_on_timeout=" << info.term.to_s if info.term
       s << " state="   << info.state.to_s   if info.state
       s
     end
   end
-
 end

data/lib/rack/timeout/support/monotonic_time.rb

@@ -25,5 +25,4 @@ module Rack::Timeout::MonotonicTime
   when RUBY_PLATFORM == "java"           ; alias fsecs fsecs_java
   else                                   ; alias fsecs fsecs_ruby
   end
-
 end

data/lib/rack/timeout/support/scheduler.rb

@@ -151,5 +151,4 @@ class Rack::Timeout::Scheduler
   instance_methods(false).each do |m|
     define_singleton_method(m) { |*a, &b| singleton.send(m, *a, &b) }
   end
-
 end

data/lib/rack/timeout/support/timeout.rb

@@ -25,5 +25,4 @@ class Rack::Timeout::Scheduler::Timeout
   def self.timeout(secs, &block)
     (@singleton ||= new).timeout(secs, &block)
   end
-
 end

data/test/env_settings_test.rb

@@ -17,4 +17,11 @@ class EnvSettingsTest < RackTimeoutTest
     end
   end
 
+  def test_term
+    with_env(RACK_TIMEOUT_TERM_ON_TIMEOUT: 1) do
+      assert_raises(SignalException) do
+        get "/sleep"
+      end
+    end
+  end
 end

data/test/test_helper.rb

@@ -42,5 +42,4 @@ class RackTimeoutTest < Test::Unit::TestCase
   def time_in_msec(t = Time.now)
     "#{t.tv_sec}#{t.tv_usec/1000}"
   end
-
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rack-timeout
 version: !ruby/object:Gem::Version
-  version: 0.5.2
+  version: 0.6.0
 platform: ruby
 authors:
 - Caio Chassot
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-10-25 00:00:00.000000000 Z
+date: 2019-12-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -91,8 +91,8 @@ licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/sharpstone/rack-timeout/issues
-  changelog_uri: https://github.com/sharpstone/rack-timeout/blob/v0.5.2/CHANGELOG.md
-  documentation_uri: https://rubydoc.info/gems/rack-timeout/0.5.2/
+  changelog_uri: https://github.com/sharpstone/rack-timeout/blob/v0.6.0/CHANGELOG.md
+  documentation_uri: https://rubydoc.info/gems/rack-timeout/0.6.0/
   source_code_uri: https://github.com/sharpstone/rack-timeout
 post_install_message: 
 rdoc_options: []
@@ -109,8 +109,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.5.2.3
+rubygems_version: 3.0.6
 signing_key: 
 specification_version: 4
 summary: Abort requests that are taking too long