httpx 0.17.0 → 0.18.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f6b2befec2e4b0093acd45f7ef9ef448ad41516510710616aded46934f1e3981
-  data.tar.gz: a9858adfacbdc27e1097b98b958f7dd1614f1207c74ec4290a54268df6270f69
+  metadata.gz: 81b7e14071e0e1978c864c63b0c8146044f255a9b2f7d9d3e917b564fb301ffb
+  data.tar.gz: 7492d34e586d86e373f62f151ceb6962719fcec156597ff34dae4c10b8347072
 SHA512:
-  metadata.gz: 86276d59efaf3a15efe0a27fbd59bff2d005bb3bab83ee9917599854bf46eba1bfbb016b9df55172c41799d1fe82195e4bb7d82c008c1996814ec2e393be71b3
-  data.tar.gz: bd113e65cf5700f231992bb485f6c59511f114372d53078ea53c019a654e449e5012b2a1a19f889d79cee1eecd24345d3e03f3f9fa50aa6c69060138327e1d09
+  metadata.gz: 5c5930cd68b2ba0f42ea60e6dc5a76ff24b2643f16c300b778e8af62f0f04d8c01baadbf0163e2b925275bbfdf44d8dd73bbf78e06dd2c86b2e54d0eb99cbbdc
+  data.tar.gz: 20a70c6ffc8895c7b54f84f9832bc8831942048ba37dc83907e0b4fef83484d1f6797a0792b075539cb985a8df65798085f3089e6746d1246e9d27ef0cb63d96

data/README.md

@@ -45,7 +45,7 @@ response = HTTPX.get("https://nghttp2.org")
 puts response.status #=> 200
 body = response.body
 puts body #=> #<HTTPX::Response ...
-``` 
+```
 
 You can also send as many requests as you want simultaneously:
 
@@ -79,7 +79,7 @@ In Ruby, HTTP client implementations are a known cheap commodity. Why this one?
 
 ### Concurrency
 
-This library supports HTTP/2 seamlessly (which means, if the request is secure, and the server support ALPN negotiation AND HTTP/2, the request will be made through HTTP/2). If you pass multiple URIs, and they can utilize the same connection, they will run concurrently in it. 
+This library supports HTTP/2 seamlessly (which means, if the request is secure, and the server support ALPN negotiation AND HTTP/2, the request will be made through HTTP/2). If you pass multiple URIs, and they can utilize the same connection, they will run concurrently in it.
 
 However if the server supports HTTP/1.1, it will try to use HTTP pipelining, falling back to 1 request at a time if the server doesn't support it (if the server support Keep-Alive connections, it will reuse the same connection).
 
@@ -137,7 +137,8 @@ In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyry
 
 ### Known bugs
 
-Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryderchuck/httpx/issues/36)).
+* Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryderchuck/httpx/issues/36)).
+* Using `total_timeout` along with the `:persistent` plugin [does not work as you might expect](https://gitlab.com/honeyryderchuck/httpx/-/wikis/Timeouts#total_timeout).
 
 ## Contributing
 

data/doc/release_notes/0_18_0.md

@@ -0,0 +1,69 @@
+# 0.18.0
+
+## Features
+
+### Response Cache
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Response-Cache
+
+The `:response_cache` plugin handles transparent usage of HTTP caching and conditional requests to improve performance and bandwidth usage.
+
+```ruby
+client = HTTPX.plugin(:response_cache)
+r1 = client.get("https://nghttp2.org/httpbin/cache")
+r2 = client.get("https://nghttp2.org/httpbin/cache")
+
+r1.status #=> 200
+r2.status #=> 304
+r1.body == r2.body #=> true
+```
+
+### jitter on "retry-after"
+
+On the `:retries` plugin, jitter calculation is now applied to the value in seconds defined by user after which a request should be retried (i.e. if `:retry_after` option is set to `2`, the retry interval may be `1.5422312` seconds, for example). This is important to avoid cases of synchronized "thundering herd", where server rejects requests, but they all get retried at the same time because the retry interval is exactly the same.
+
+You can override the jitter calculation function by using the [:retry_jitter](https://gitlab.com/honeyryderchuck/httpx/-/wikis/Retries#retry_jitter) option:
+
+```ruby
+HTTPX.plugin(:retries, retry_after: 2, retry_jitter: ->(interval) { interval + rand }) # interval is 3
+```
+
+You can opt out of this by setting `HTTPX_NO_JITTER=1` environment variable.
+
+### Response#error
+
+`HTTPX::Response#error` was added, to match `HTTPX::Response#error`. It returns an exception for 4xx/5xx responses (`HTTPX::HTTPError`), `nil` otherwise. It allows for end users to write such code:
+
+```ruby
+if (response = HTTPX.get(uri)).error
+  # success
+else
+  #error
+end
+```
+
+## Improvements
+
+* `webmock` adapter: added support for "stub_http_request#to_timeout" (https://gitlab.com/honeyryderchuck/httpx/-/merge_requests/165).
+
+## timers not a dependency
+
+The functionality provided by the `timers` gem was replaced by a simpler custom implementation. Although powerful, its complexity was somewhat unnecessary for `httpx`'s simpler event loop, where user-defined timeouts are usually the same for a given batch of requests. The removal of `timers` reduces the number of dependencies to 1, which is `http-2-next` and is maintained by me.
+
+## AWS plugins
+
+* `aws_sdk_authentication` plugin: removed implementation relying on  `aws-sdk-s3`, replacing it with an `aws-sdk-core` relying implementation, which only uses credentials strategies and region discovery (the whole point of this SDK is to use a minimal subset of AWS SDK).
+
+
+## Bugfixes
+
+* Fixed Error class declaration on response decoders when mime type is invalid (https://gitlab.com/honeyryderchuck/httpx/-/merge_requests/166).
+* `ErrorResponse#to_s` now removes ANSI escape sequences from error backtraces.
+* Persistent connections were kept around both in the pool and in the selector; the first is necessary, but the second caused busy loop scenarios all over; they are now removed when no requests are being handled.
+* Connections which failed connection handshake were removed from the pool, but not from the selector list, causing busy loop scenarios in a few cases; this has been fixed.
+* Fixed issue where HTTP/2 streams were being closed twice (and signaling it also twice), messing connection accounting in the pool.
+* DoH resolver was always subscribed to the default thread "connection pool", which broke scenarios were session was patched to use its custom pool; it now ensures that it subscribes to the pool it was created in.
+* `:aws_sigv4` plugin: removed require of `aws-sdk-s3`, left there by mistake (the whole point of the plugin is to run without the AWS SDK).
+## Chore
+
+* `HTTPX::ErrorResponse#status` is now deprecated.

data/doc/release_notes/0_18_1.md

@@ -0,0 +1,12 @@
+# 0.18.1
+
+## Bugfixes
+
+* HTTP/1.1 pipelining logs were logging the previously-buffered requests all together for each triggered request, which created some confusion for users when reporting errors. This has been fixed.
+* HTTP/2 coalescing is now skipped when performing TLS connections with VERIFY_NONE.
+* HTTP/2 peer GOAWAY frames will now result in a (retryable) connection error, instead of being ignored and leaving a "ghost" connection behind.
+* fixed total timeout call which was not raising the exception.
+
+## Chore
+
+This gem now requires MFA-based gem releases.

data/doc/release_notes/0_18_2.md

@@ -0,0 +1,10 @@
+# 0.18.2
+
+## Bugfixes
+
+* A bug was reported and fixed, whereby a persistent connection with a `:total_timeout` set was triggering the timeout and leaving the process looping indefinitely.
+
+
+## Chore
+
+The quirk of using the `:persistent` plugin with `:total_timeout` has been documented: https://gitlab.com/honeyryderchuck/httpx/-/wikis/Timeouts#total_timeout.

data/doc/release_notes/0_18_3.md

@@ -0,0 +1,7 @@
+# 0.18.3
+
+## Bugfixes
+
+* request bodies eager-loaded from enumerables yield duped partial chunks.
+
+An error was observed while looking at webmock integration, where requests formed via the multipart plugin where returning an empty string as body. The issue was caused by an optimization on multipart encoder, which reuses the same buffer when reading chunks. Unfortunately, these cannot be yielded the same way via IO.copy_stream, as the same (cleared) buffer will be used to generate the eager-loaded body chunks.

data/lib/httpx/adapters/datadog.rb

@@ -64,7 +64,7 @@ module Datadog
           def finish(response)
             return unless @span
 
-            if response.respond_to?(:error)
+            if response.is_a?(::HTTPX::ErrorResponse)
               @span.set_error(response.error)
             else
               @span.set_tag(Datadog::Ext::HTTP::STATUS_CODE, response.status.to_s)

data/lib/httpx/adapters/faraday.rb

@@ -22,6 +22,8 @@ module Faraday
       # :nocov:
 
       module RequestMixin
+        using ::HTTPX::HashExtensions
+
         private
 
         def build_request(env)
@@ -38,7 +40,7 @@ module Faraday
           timeout_options = {
             connect_timeout: env.request.open_timeout,
             operation_timeout: env.request.timeout,
-          }.reject { |_, v| v.nil? }
+          }.compact
 
           options = {
             ssl: {},
@@ -101,7 +103,7 @@ module Faraday
           end
 
           def on_response(&blk)
-            if block_given?
+            if blk
               @on_response = lambda do |response|
                 blk.call(response)
               end
@@ -112,7 +114,7 @@ module Faraday
           end
 
           def on_complete(&blk)
-            if block_given?
+            if blk
               @on_complete = blk
               self
             else

data/lib/httpx/adapters/webmock.rb

@@ -86,7 +86,9 @@ module WebMock
         end
 
         def _build_from_webmock_response(request, webmock_response)
-          return ErrorResponse.new(request, webmock_response.exception, request.options) if webmock_response.exception
+          return _build_error_response(request, HTTPX::TimeoutError.new(1, "Timed out")) if webmock_response.should_timeout
+
+          return _build_error_response(request, webmock_response.exception) if webmock_response.exception
 
           response = request.options.response_class.new(request,
                                                         webmock_response.status[0],
@@ -95,6 +97,10 @@ module WebMock
           response << webmock_response.body.dup
           response
         end
+
+        def _build_error_response(request, exception)
+          HTTPX::ErrorResponse.new(request, exception, request.options)
+        end
       end
     end
 

data/lib/httpx/altsvc.rb

@@ -10,14 +10,14 @@ module HTTPX
     module_function
 
     def cached_altsvc(origin)
-      now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      now = Utils.now
       @altsvc_mutex.synchronize do
         lookup(origin, now)
       end
     end
 
     def cached_altsvc_set(origin, entry)
-      now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      now = Utils.now
       @altsvc_mutex.synchronize do
         return if @altsvcs[origin].any? { |altsvc| altsvc["origin"] == entry["origin"] }
 

data/lib/httpx/chainable.rb

@@ -4,9 +4,9 @@ module HTTPX
   module Chainable
     %i[head get post put delete trace options connect patch].each do |meth|
       class_eval(<<-MOD, __FILE__, __LINE__ + 1)
-        def #{meth}(*uri, **options)
-          request(:#{meth}, uri, **options)
-        end
+        def #{meth}(*uri, **options)        # def get(*uri, **options)
+          request(:#{meth}, uri, **options) #   request(:get, uri, **options)
+        end                                 # end
       MOD
     end
 

data/lib/httpx/connection/http1.rb

@@ -36,6 +36,8 @@ module HTTPX
 
       request = @requests.first
 
+      return unless request
+
       return :w if request.interests == :w || !@buffer.empty?
 
       :r
@@ -283,10 +285,10 @@ module HTTPX
         # on the last request of the possible batch (either allowed max requests,
         # or if smaller, the size of the batch itself)
         requests_limit = [@max_requests, @requests.size].min
-        if request != @requests[requests_limit - 1]
-          "keep-alive"
-        else
+        if request == @requests[requests_limit - 1]
           "close"
+        else
+          "keep-alive"
         end
       end
 
@@ -313,8 +315,9 @@ module HTTPX
     end
 
     def join_headers(request)
-      @buffer << "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}" << CRLF
-      log(color: :yellow) { "<- HEADLINE: #{@buffer.to_s.chomp.inspect}" }
+      headline = "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}"
+      @buffer << headline << CRLF
+      log(color: :yellow) { "<- HEADLINE: #{headline.chomp.inspect}" }
       extra_headers = set_protocol_headers(request)
       join_headers2(request.headers.each(extra_headers))
       log { "<- " }

data/lib/httpx/connection/http2.rb

@@ -16,6 +16,12 @@ module HTTPX
       end
     end
 
+    class GoawayError < Error
+      def initialize
+        super(0, :no_error)
+      end
+    end
+
     attr_reader :streams, :pending
 
     def initialize(buffer, options)
@@ -291,16 +297,18 @@ module HTTPX
     end
 
     def on_stream_refuse(stream, request, error)
-      stream.close
       on_stream_close(stream, request, error)
+      stream.close
     end
 
     def on_stream_close(stream, request, error)
+      return if error == :stream_closed && !@streams.key?(request)
+
       log(level: 2) { "#{stream.id}: closing stream" }
       @drains.delete(request)
       @streams.delete(request)
 
-      if error && error != :no_error
+      if error
         ex = Error.new(stream.id, error)
         ex.set_backtrace(caller)
         response = ErrorResponse.new(request, ex, request.options)
@@ -342,9 +350,16 @@ module HTTPX
 
     def on_close(_last_frame, error, _payload)
       is_connection_closed = @connection.state == :closed
-      if error && error != :no_error
+      if error
         @buffer.clear if is_connection_closed
-        ex = Error.new(0, error)
+        if error == :no_error
+          ex = GoawayError.new
+          @pending.unshift(*@streams.keys)
+          @drains.clear
+          @streams.clear
+        else
+          ex = Error.new(0, error)
+        end
         ex.set_backtrace(caller)
         handle_error(ex)
       end
@@ -388,10 +403,10 @@ module HTTPX
     end
 
     def on_pong(ping)
-      if !@pings.delete(ping.to_s)
-        close(:protocol_error, "ping payload did not match")
-      else
+      if @pings.delete(ping.to_s)
         emit(:pong)
+      else
+        close(:protocol_error, "ping payload did not match")
       end
     end
   end

data/lib/httpx/connection.rb

@@ -70,7 +70,7 @@ module HTTPX
 
       @inflight = 0
       @keep_alive_timeout = @options.timeout[:keep_alive_timeout]
-      @keep_alive_timer = nil
+      @total_timeout = @options.timeout[:total_timeout]
 
       self.addresses = @options.addresses if @options.addresses
     end
@@ -117,7 +117,8 @@ module HTTPX
     def coalescable?(connection)
       if @io.protocol == "h2" &&
          @origin.scheme == "https" &&
-         connection.origin.scheme == "https"
+         connection.origin.scheme == "https" &&
+         @io.can_verify_peer?
         @io.verify_hostname(connection.origin.host)
       else
         @origin == connection.origin
@@ -200,11 +201,9 @@ module HTTPX
     end
 
     def close
-      @parser.close if @parser
-      return unless @keep_alive_timer
+      transition(:active) if @state == :inactive
 
-      @keep_alive_timer.cancel
-      remove_instance_variable(:@keep_alive_timer)
+      @parser.close if @parser
     end
 
     def reset
@@ -216,26 +215,40 @@ module HTTPX
     def send(request)
       if @parser && !@write_buffer.full?
         request.headers["alt-used"] = @origin.authority if match_altsvcs?(request.uri)
-        if @keep_alive_timer
+
+        if @response_received_at && @keep_alive_timeout &&
+           Utils.elapsed_time(@response_received_at) > @keep_alive_timeout
           # when pushing a request into an existing connection, we have to check whether there
           # is the possibility that the connection might have extended the keep alive timeout.
           # for such cases, we want to ping for availability before deciding to shovel requests.
-          if @keep_alive_timer.fires_in.negative?
-            @pending << request
-            parser.ping
-            return
-          end
-
-          @keep_alive_timer.pause
+          @pending << request
+          parser.ping
+          transition(:active) if @state == :inactive
+          return
         end
-        @inflight += 1
-        parser.send(request)
+
+        send_request_to_parser(request)
       else
         @pending << request
       end
     end
 
     def timeout
+      if @total_timeout
+        return @total_timeout unless @connected_at
+
+        elapsed_time = @total_timeout - Utils.elapsed_time(@connected_at)
+
+        if elapsed_time.negative?
+          ex = TotalTimeoutError.new(@total_timeout, "Timed out after #{@total_timeout} seconds")
+          ex.set_backtrace(caller)
+          on_error(ex)
+          return
+        end
+
+        return elapsed_time
+      end
+
       return @timeout if defined?(@timeout)
 
       return @options.timeout[:connect_timeout] if @state == :idle
@@ -243,6 +256,14 @@ module HTTPX
       @options.timeout[:operation_timeout]
     end
 
+    def deactivate
+      transition(:inactive)
+    end
+
+    def open?
+      @state == :open || @state == :inactive
+    end
+
     private
 
     def connect
@@ -379,9 +400,7 @@ module HTTPX
 
     def send_pending
       while !@write_buffer.full? && (request = @pending.shift)
-        @inflight += 1
-        @keep_alive_timer.pause if @keep_alive_timer
-        parser.send(request)
+        send_request_to_parser(request)
       end
     end
 
@@ -389,6 +408,15 @@ module HTTPX
       @parser ||= build_parser
     end
 
+    def send_request_to_parser(request)
+      @inflight += 1
+      parser.send(request)
+
+      return unless @state == :inactive
+
+      transition(:active)
+    end
+
     def build_parser(protocol = @io.protocol)
       parser = registry(protocol).new(@write_buffer, @options)
       set_parser_callbacks(parser)
@@ -400,7 +428,8 @@ module HTTPX
         AltSvc.emit(request, response) do |alt_origin, origin, alt_params|
           emit(:altsvc, alt_origin, origin, alt_params)
         end
-        handle_response
+        @response_received_at = Utils.now
+        @inflight -= 1
         request.emit(:response, response)
       end
       parser.on(:altsvc) do |alt_origin, origin, alt_params|
@@ -420,7 +449,7 @@ module HTTPX
       end
       parser.on(:close) do |force|
         transition(:closing)
-        if force
+        if force || @state == :idle
           transition(:closed)
           emit(:close)
         end
@@ -435,6 +464,7 @@ module HTTPX
           transition(:closing)
           transition(:closed)
           emit(:reset)
+
           @parser.reset if @parser
           transition(:idle)
           transition(:open)
@@ -466,15 +496,17 @@ module HTTPX
       when :open
         return if @state == :closed
 
-        total_timeout
-
         @io.connect
         return unless @io.connected?
 
+        @connected_at = Utils.now
+
         send_pending
 
         @timeout = @current_timeout = parser.timeout
         emit(:open)
+      when :inactive
+        return unless @state == :open
       when :closing
         return unless @state == :open
 
@@ -482,15 +514,15 @@ module HTTPX
         return unless @state == :closing
         return unless @write_buffer.empty?
 
-        if @total_timeout
-          @total_timeout.cancel
-          remove_instance_variable(:@total_timeout)
-        end
-
         purge_after_closed
       when :already_open
         nextstate = :open
         send_pending
+      when :active
+        return unless @state == :inactive
+
+        nextstate = :open
+        emit(:activate)
       end
       @state = nextstate
     rescue Errno::ECONNREFUSED,
@@ -506,44 +538,24 @@ module HTTPX
     def purge_after_closed
       @io.close if @io
       @read_buffer.clear
-      if @keep_alive_timer
-        @keep_alive_timer.cancel
-        remove_instance_variable(:@keep_alive_timer)
-      end
-
       remove_instance_variable(:@timeout) if defined?(@timeout)
     end
 
-    def handle_response
-      @inflight -= 1
-      return unless @inflight.zero?
-
-      if @keep_alive_timer
-        @keep_alive_timer.resume
-        @keep_alive_timer.reset
-      else
-        @keep_alive_timer = @timers.after(@keep_alive_timeout) do
-          unless @inflight.zero?
-            log { "(#{@origin}): keep alive timeout expired" }
-            parser.ping
-          end
-        end
-      end
-    end
-
     def on_error(error)
       if error.instance_of?(TimeoutError)
-        if @timeout
-          @timeout -= error.timeout
-          return unless @timeout <= 0
-        end
 
-        if @total_timeout && @total_timeout.fires_in.negative?
-          ex = TotalTimeoutError.new(@total_timeout.interval, "Timed out after #{@total_timeout.interval} seconds")
+        if @total_timeout && @connected_at &&
+           Utils.elapsed_time(@connected_at) > @total_timeout
+          ex = TotalTimeoutError.new(@total_timeout, "Timed out after #{@total_timeout} seconds")
           ex.set_backtrace(error.backtrace)
           error = ex
-        elsif connecting?
-          error = error.to_connection_error
+        else
+          if @timeout
+            @timeout -= error.timeout
+            return unless @timeout <= 0
+          end
+
+          error = error.to_connection_error if connecting?
         end
       end
       handle_error(error)
@@ -558,18 +570,5 @@ module HTTPX
         request.emit(:response, response)
       end
     end
-
-    def total_timeout
-      total = @options.timeout[:total_timeout]
-
-      return unless total
-
-      @total_timeout ||= @timers.after(total) do
-        ex = TotalTimeoutError.new(total, "Timed out after #{total} seconds")
-        ex.set_backtrace(caller)
-        on_error(ex)
-        @parser.close if @parser
-      end
-    end
   end
 end

data/lib/httpx/domain_name.rb

@@ -123,7 +123,7 @@ module HTTPX
 
       # RFC 6265 #4.1.1
       # Domain-value must be a subdomain.
-      @domain && self <= domain && domain <= @domain ? true : false
+      @domain && self <= domain && domain <= @domain
     end
 
     # def ==(other)